Files retrieval from a index

[DigitalMaxim]

Hello,

When I am running the command -
python manage.py runscript -e snugglefish_service index -- -a getnew -n FOO -c 500000 | xargs -I % -n 1 python manage.py runscript -e crits_scripts get_file -- -m % -o /home/wxs/FOO/%

I am receiving the list of the hashes as a screen output, but the actual directory /home/wxs/FOO remains empty.
What might be a cause of this?
Also it seems that -e option not working for me, every time that I run this command it requires authentication.

Thank you.

[wxsBSD]

It sounds like the hashes are printed to something other than stdout (likely stderr). Can you try redirecting stderr to stdout in the first part of the pipeline? Something like this:

python manage.py runscript -e snugglefish_service index -- -a getnew -n FOO -c 500000 2>&1 | xargs -I % -n 1 python manage.py runscript -e crits_scripts get_file -- -m % -o /home/wxs/FOO/%

The -e option to runscript uses environment variables for authentication. If they are not set then it falls back to prompting for authentication to CRITs. Please take a look at python manage.py runscript -h.

[DigitalMaxim]

After adding 2>&1 it returns error:
CommandError: No module named crits_scripts.scripts.get_file

Also I didn't mention that while running this command I've received this output:
fatal: Not a git repository (or any parent up to mount point /mnt/crits)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set)

[DigitalMaxim]

I do have write permissions to the output directory but still nothing saved in it. Can you please advise?
Thank you.

[wxsBSD]

Have you configured CRITs to use services in where ever you put your them? Most people put them in /data/crits_services and then configure CRITs to use that directory for services.

[DigitalMaxim]

First of all I'd like to thank you for your assistance. I am really appreciate it.

I've configured CRITs to use services, but I didn't put these services at the root directory. At the CRITs GUI all services are displaying as available, including snugglefish service. I just can't use snugglefish service because I am not succeeding to finish indexing of the sample files we've uploaded so far.

Is there any log that I can check to see why it is not saving files at the given directory?

Thanks

[wxsBSD]

I can dig into this a bit on Monday when I'm back in the office.

[DigitalMaxim]

Great, looking forward to hear any news from you.
Thanks,

[wxsBSD]

Can you provide more information about the various directories involved and where you are executing things from. Everything you are doing seems more or less right but it's not clear exactly what you are doing. A direct copy/paste of the commands and the exact output would help, along with a description of where your services live. In particular the error: CommandError: No module named crits_scripts.scripts.get_file is indicative of a problem finding the crits_scripts directory in the services. This makes me think you don't have the entire services repository in place, and have selectively chosen to exclude crits_scripts.

[DigitalMaxim]

Hi,

The CRITs directory is /mnt/crits/crits-m
All CRITs services are at /mnt/crits/data/crits_services

The first thing that I did while starting to index for snugglefish is:

python manage.py runscript snugglefish_service index -- -a create -n CW -q "{'source.name': 'CW'}" -d /mnt/crits/data/crits_services/snugglefish_service/index

Then, I am running this command to check the status:
python manage.py runscript snugglefish_service index -- -a status -n CW
The output I am receiving:
fatal: Not a git repository (or any parent up to mount point /mnt/crits)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Username:
Password:
Name: CW
Directory: /mnt/crits/data/crits_services/snugglefish_service/index/
Created: 2015-06-02 08:02:01.343000
Last update: 2015-06-02 08:02:01.343000
Query: {'source.name': 'CW'}
Last ID: 556f0233780f5256a4b89921
Total objects: 26
Count indexed: 0
Percent indexed: 0.000000
It looks exactly as an example at Snugglefish_service page at Github.

Then I am trying to get first 500 files for the CW index:
python manage.py runscript -e snugglefish_service index -- -a getnew -n CW -c 500 | xargs -I % -n 1 python manage.py runscript -e crits_scripts get_file -- -m % -o /mnt/crits/data/crits_services/snugglefish_service/wxs/CW/%

The output that I see on the screen:
fatal: Not a git repository (or any parent up to mount point /mnt/crits)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Username:
Password:
d2f825ecfb3d979950b9de92cbe29286
7a83f701287f7624023f2a95fe37a2e6
254ff246f7603d54990f1894b2f129e
and some more hashes...
Total number of hashes 26 , exactly as a total objects in the CW index.

But when I am checking the directory mnt/crits/data/crits_services/snugglefish_service/wxs/CW/ , it is empty.
This the point that I am stuck and can't figure out what I am doing wrong.

[wxsBSD]

So the hashes are being printed to stderr (I still need to figure out how to get scripts to properly print to stdout instead of stderr). You can get the hashes to stdout by simply redirecting stderr to stdout, as I suggested earlier. Do this by executing:

python manage.py runscript -e snugglefish_service index -- -a getnew -n CW -c 500 2>&1 | xargs -I % -n 1 python manage.py runscript -e crits_scripts get_file -- -m % -o /mnt/crits/data/crits_services/snugglefish_service/wxs/CW/%

Notice I am adding 2>&1 to the end of the first part of the pipeline. This is what you did before but said you were getting a CommandError: No module named crits_scripts.scripts.get_file message. What I want to make sure is that /mnt/crits/data/crits_services/crits_scripts/scripts/get_file.py exists.

[DigitalMaxim]

Hi,

I've run this command:
python manage.py runscript -e snugglefish_service index -- -a getnew -n CW -c 500 2>&1 | xargs -I % -n 1 python manage.py runscript -e crits_scripts get_file -- -m % -o /mnt/crits/data/crits_services/snugglefish_service/wxs/CW/%

Output on the screen:
fatal: Not a git repository (or any parent up to mount point /mnt/crits)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Username: fatal: Not a git repository (or any parent up to mount point /mnt/crits)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Username:
Password:
I've entered my username and password and it not returned any error but just stuck at this point.
After waiting for about 10 minutes, I've cancelled it by Ctrl+C and received the following:
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(_args, *_options.dict)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 285, in execute
output = self.handle(_args, *_options)
File "/mnt/crits/crits-master/crits/core/management/commands/runscript.py", line 64, in handle
username = getpass.getpass("Username: ")
File "/usr/lib/python2.7/getpass.py", line 71, in unix_getpass
passwd = _raw_input(prompt, stream, input=input)
File "/usr/lib/python2.7/getpass.py", line 133, in _raw_input
line = input.readline()

Thanks,

[wxsBSD]

So the -e argument to runscript is used to get CRITs authentication information from the environment. You should set CRITS_USER and CRITS_PASSWORD in your environment and it won't ask you for that information. In this particular case it looks like stdin was being gobbled up.

Read this for more information: https://github.com/crits/crits/wiki/runscript

[DigitalMaxim]

Added these environments with "export" command and when I do "printenv" I see CRITS_USER=my user CRITS_PASSWORD= my password , but seems that it doesn't work.
Also I just created an empty .git directory to resolve this error - fatal: Not a git repository (or any parent up to mount point /mnt/crits) It returns new error now : fatal: Needed a single revision.

Can you please advise what's wrong here also?
Thanks

[DigitalMaxim]

Hello,

I have additional question. Every time that I am uploading new samples to the CRITs should I delete the existing index and create a new one? Is there any way just to update the existing index with a new sample files?
Also is there any way to reset the "count" back to the 0? Every time that I ran this command python /mnt/crits/crits-master/manage.py runscript -e snugglefish_service index -- -a update -n CW -c $(ls /mnt/crits/data/crits_services/snugglefish_service/wxs/CW/ | wc -l) it doubles the count number. Is this normal and expected?

Thank you,