Troubleshooting TMSP

1. Troubleshooting TMSP Marks Shen • Senior Engineer – QAEvan Wang • Engineer - QA

2. Agenda | • Frequent Case • Debug log and information • Troubleshooting • Q&A 2 Classification 11/8/2014

3. No report was generated • Logs need to collect on Daemon Server • /opt/TrendMicro/tdss/tdes/log/iae_log.txt • /var/log/cron • Information • Customer expiration date • Device register to TMSP

4. No report was generated Cont. • Normal debug log for log correlation and report generation • Crontab task cron_iae.sh will be executed at 2:15 am every day

5. No report was generated Cont. Report will not be generated if • Customer service get expired • Customer without device registered

6. No incident in report • Logs need to collect on Daemon Server • /opt/TrendMicro/tdss/tdes/log/iae_log.txt • Information • Check if TDA log has uploaded Latest log time

7. Cannot access Admin console (err 404) • Logs need to collect on Daemon Server • /var/log/httpd/access_log • /var/log/httpd/error_log • Information • ps –ef | grep httpd • Netstat –anp | grep httpd

8. No Rsync log uploaded • Logs need to collect on Access Server • Log receiver • /home/tdalog/log/pre-post-exec.log /home/tdalog/log/db_import_tda.log • /home/tdalog/log/db_import_tdm.log • Authentication (describe in next sides) • Information • Ps –ef | grep tmsshd • Netstat –anp | grep tmsshd • Listen on port 22

9. No Rsync log uploaded Cont. • Normal procedure debug log of TDA log processing • /home/tdalog/log/pre-post-exec.log

10. CAS server case Problems caused by CAS failure: • Device register to TMSP fail • Customer portal login fail • Log uploading fail through RSYNC Logs need to collect on Access Server: • /var/log/messages • /var/log/cas_8000.log • /var/log/cas_8001.log • /var/log/cas_8002.log Information: • Ps –ef | grep pound • Ps –ef | grep rubcasd Normal log of CAS authentication • /var/log/cas_8000.log

11. CAS server case – Service down • TDA register fail • Check /var/log/messages • Recover • If pound or cas service is down • /etc/init.d/pound start • /etc/init.d/rubcasd start

12. Data Gateway Case Problems caused by Data Gateway failure: • OCS Heartbeat / OCS log real-time cannot upload • T2 / T3 mitigation request cannot deliver to TMTM • SIC sample cannot upload from TMTM Logs need to collect on Access Server: • /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/dg/WEB-INF/logs/dg.log • $APACHE_HOME/logs/ssl_request_log_dg • $APACHE_HOME/logs/error_log • /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/catalina.out Information: • Ps –ef | grep httpd • Ps –ef | grep tomcat • Netstat –anp | grep 443 • Netstat –anp | grep 8009

13. Data Gateway Case – DB disconnect • TDA register fail • Check apache error log: /usr/apache/logs/error_log

14. Customer portal cannot login • Logs need to collect on Access Server: • /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/tms2/WEB-INF/logs/tms.log • $APACHE_HOME/logs/ssl_request_log_portal • $APACHE_HOME/logs/error_log • /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/catalina.out • Information • Ps –ef | grep httpd • Ps –ef | grep tomcat • Netstat –anp | grep 443 • Netstat –anp | grep 8009

15. Cannot get eMail notification • Exclude Mail server problem, collect debug logs: • /root/infomation.log • /var/log/cron No Subscription DB connection fail

16. FAQ

17. FAQ Q: How to re-generate report manually? Login TDES machine, change dir to "/opt/TrendMicro/tdss/tdes/"; • 1. Daily Report #php gendailydata.php user_id YEAR MONTH DAY example: "php gendailydata.php trend 2009 01 04" generate daily report of 2009.01.04 for customer "trend“ • 2. Executive Report (Weekly / Monthly) #php genexecdata.php user_id START_DATE START_DATE yes m/w START_DATE: report start data with format “YYYY-MM-DD” START_DATE: report end data with format “YYYY-MM-DD” yes: if this report will be imported into DB m/w: monthly or weekly example: "php genexecdata.php trend 2009-01-01 2009-01-31 yes m" to generate monthly report for customer "trend" of 2009-01 • 3. Upsell Report # php genupselldata.php user_id START_DATE END_DATE no example: " php genupselldata.php trend 2009-01-01 2009-01-31 no" to generate upsell report for customer "trend" from 2009-01-01 to 2009-01-31 Note: Upsell report will not be imported into DB and cannot download from admin console

18. FAQ Q:What the difference between two service type “TDS” and “TLMS”? A:“TDS” service is set to the customer, who has TDA deployed. “TLMS” service is set to the customer, who has both TDA and TMTM deployed. TMSP provide different report for these 2 service type, and for TLMS, TMSP provide additional UI for abnormal endpoints handling process.

19. FAQ Q: Why AE status show N/A on Admin console? A: AE (Abnormal Endpoint) status is designed for the customers, who has TMTM deployed, to show the mitigation failed client on TMSP UI. That means, only the customer with service type “TLMS” will show the AE details in TMSP admin UI as link.

20. FAQ Q: Whether the IAE will be updated? A:Yes, TMSP IAE rules will be updated timely to detect new threats. Currently, the rule is updated monthly. In TMSP side, we get new threat rule from threat team and deploy to each site after verification.

21. FAQ Q:Can TMSP generate DOC format report? A:No, currently, TMSP can only generate report in PDF format.

22. FAQ Q: How to backup Database? A: Here are two type of DB backup, Full Backup and Lightweight Backup. • Full backup: Backup full database of TMSP #mysqldump -uroot -p"$password" tdors_new > tdors_new_backup.sql

23. FAQ • Lightweight DB backup: Backup customer info, configuration and some correlation result. #mysqldump -h $host -u$account -p"$password" tdors_new access_code admin_log admins contacts customer customer_expiration_conf customer_expiration_notif notification ocs_ast_event ocs_malware scsd_case scsd_case_followup scsd_case_status scsd_pattern scsd_pattern_status scsd_sample_info scsd_sample_summary t2t3_dce_result tb_devprofile> SJDC2.5.sql After backup entire database, it is recommended to store the backup file into another physical storage.

24. Q&A

25. THANK YOU! 25 Classification 11/8/2014