1 / 9

Qt 语义一致性大赛

Qt 语义一致性大赛. Yue Lin (linyue29@gmail.com) Work at Guangzhou (2010.9 – 2013.3) Computer Science Department, Zhejiang University Supervisor: Xiaofei He, Deng Cai. Stage 1:. 分词工具选择： 结巴中文分词 中科院分词系统 ICTCLAS SmallSeg SnailSeg

majed
Download Presentation

Qt 语义一致性大赛

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Qt语义一致性大赛 Yue Lin (linyue29@gmail.com) Work atGuangzhou (2010.9 – 2013.3) Computer Science Department, Zhejiang University Supervisor: Xiaofei He, Deng Cai

  2. Stage 1: • 分词工具选择： • 结巴中文分词 • 中科院分词系统ICTCLAS • SmallSeg • SnailSeg • 鉴于采用python进行数据处理，并且在一些比较中(http://www.zhihu.com/question/19578687)，结巴分词的效果还是比较优秀的，所以最终选择结巴中文分词

  3. Stage 2: • 构造停用词表 • 网上搜集后进行增删 • 对于分词后属于停用词的词语和符号，全部忽略

  4. Stage 3: • 得到词库表，并且赋予每个词一个weight • 采用传统的TF-IDF算法，利用开源的搜狗实验室的语料库，进行分词统计处理后，得到每个词的权重 • 对于不在库中的词语，我们根据该词的词性赋予相应的权重

  5. Stage 4: • 采用的3种计算Similarity的算法： • Vectorial Model and Cosine Similarity • Jacard Similarity • Ngram Similarity

  6. Stage 5： • 如何判断两个词语是否相近 • 曾经尝试使用google开源的基于Deep Learning的word2vector库，利用搜狗语料库进行训练，但是效果很差，所以最后采用了简单的判断两个词的词性以及重合的字数进行近似的判断，效果有了一定的提升。

  7. Stage 6： • 特征融合 • 用LibSVM把3种相似度作为特征进行融合，得到相应的模型系数后，调整阈值使得F值在training集和test集都有不错的performance。

  8. 后续讨论： • Case：叔叔的妈妈叫什么？ VS. 妈妈的叔叔叫什么？ • Solution：Shallow SOV parsing？ • Case: W262充电 VS. W262不充电 • Solution: 加大否定词的权重？

  9. Thank You!

More Related