Collect and analyze Ptt discussions about beauty products to assist marketing staff to understand user reviews of the products.
◆ Custom search keywords: Create various combinations of keywords by users. For example, users can set brand and multiple product keywords and connect them by different conditions.
◆ Automated web crawler: Use cheerio to crawl websites and crontab to active web crawler on schedule.
◆ Data pipeline: Established an automatic process including web crawler, unifying data format, deleting repeated data, and storing data on schedule.
◆ Search for articles: Use MySQL(in AWS RDS) to store Ptt discussions and search related data in MySQL by using custom search keywords.
◆ Ratio of positive and negative reviews: Use Google Natural Language API to identify emotions of articles.
◆ Monitoring negative reviews and alert: Store negative reviews in RDS and alert users when they log in.
◆ Word cloud: Use nodejieba to acquire the most popular words. Use wordcloud2 for data visualization.
Testing: Use Mocha to implement unit test.