CWB(The IMS Open Corpus Workbench) : managing and querying large text corpora
I will use CWB for preprocessing of topic modeling
(ubuntu 14.04.03)
1. Download CWB, extract
http://cwb.sourceforge.net/download.php
2. Install on target directory
$ sudo ./install-cwb.sh
3. Finish
4. If you want to test your CWB installation, download and unpack one of the pre-indexed demo corpora(novels by Charles Dickens)
http://cwb.sourceforge.net/download.php
5. Move to DemoCorpus/ and typing this on command line
$ cwb-describe-corpus -r registry -s DICKENS
result
it prints some information about the corpus and its attributes
6. Next type in
$ cqp -eC -r registry -D DICKENS
start an interactive CQP session and activate the demo corpus DICKENS for queries.
7. Type in some words like this, and check result.
DICKENS> "the"
result
8. Try applying options
DICKENS> set Context 1 s;
DICKENS> "the";
DICKENS> show +lemma
DICKENS> show +pos;
9. Finished settings and try testing again
****
If you want to use CWB wherever, not specific directory
1. On DemoCorpus/
$ sudo mkdir /usr/local/share/cwb
$ sudo mkdir /usr/local/share/cwb/registry
2. Open 'dickens' file in /DemoCorpus/registry and change
HOME data => HOME /home/won/Desktop/TopicModeling/CWB/DemoCorpus/data
3. Testing on other directory
$ cd registry/
$ sudo cp dickens /usr/local/share/cwb/registry/
****
CWB was written by perl. If you want to use on python, need to install python wrapper(like yannick-cwb-python)
1. Download source and unpack
https://bitbucket.org/yannick/cwb-python/src
2. Install wrapper
$ python setup.py build
In my case, error issued in this step. like that
running build
running build_py
running build_ext
building 'CWB.CL' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Isrc -I/usr/include/python2.7 -c src/CWB/CL.cpp -o build/temp.linux-x86_64-2.7/src/CWB/CL.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
src/CWB/CL.cpp: In function ‘int __pyx_pf_3CWB_2CL_6IDList___cinit__(PyObject*, PyObject*, PyObject*)’:
src/CWB/CL.cpp:1659:7: warning: variable ‘__pyx_v_is_sorted’ set but not used [-Wunused-but-set-variable]
int __pyx_v_is_sorted;
^
c++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/src/CWB/CL.o -lcl -lpcre -lglib-2.0 -o build/lib.linux-x86_64-2.7/CWB/CL.so
/usr/bin/ld: cannot find -lpcre
/usr/bin/ld: cannot find -lglib-2.0
collect2: error: ld returned 1 exit status
error: command 'c++' failed with exit status 1
just install gcc and some libraries.
$ sudo apt-get install gcc
$ sudo apt-get install libpcre3 libpcre3-dev
$ sudo apt-get install libglib2.0-dev
then install wrapper continuously
$ sudo python setup.py install
3. Confirm installation
$ python -m doctest tests/idlist.txt
which should terminate with no output when everything is well.
Ref:
http://cwb.sourceforge.net/install.php
https://bitbucket.org/yannick/cwb-python/src
