Installation

Yimeng Zhang
Feb. 29, 2016

System Requirements

For users

  • MATLAB R2012b or higher should work, although I personally only tested the program on R2012b, R2014a, and R2015b.

For developers

If you also want to run all tests and generate the doc, apart from MATLAB, a Python 2.7.x environment plus the following packages (the ones in /requirements.txt of the repository) should do.

jsl
sphinxcontrib-matlabdomain
recommonmark

How to install

Todo

verify the program on Windows.

  1. obtain the latest release of cdttable from https://github.com/leelabcnbc/cdttable by download the lastest release under https://github.com/leelabcnbc/cdttable/releases/

  2. (For Mac / Linux) Open a terminal window and install Third party libraries. This will only do some file copying and extraction inside the program directory, and won’t affect the system globally (check the scripts or see the Windows step to know what really happens). This has two steps.
    1. Change directory to /3rdparty, and run ./install_3rdparty.sh. This is required.
    2. Change directory to /3rdparty/NEV, and run ./install_NEV_3rdparty.sh. This is not necessary if you don’t want to run the demo code or use the provided +nevreader to preprocess NEV files.
  3. (For Windows) do the above step manually.
    1. copy /3rdparty/json-schema-validator-2.2.6-lib.jar to /+cdttable.
    2. In /3rdparty/NEV, copy json-schema-validator-2.2.6-lib.jar, ce_read_cortex_index.m, and ce_read_cortex_record.m to /+nevreader, and expand NPMK-4.0.0.0.zip. After this extraction, there should be a folder called NPMK in /3rdparty/NEV/, and there should be many files inside it instead of a single nested folder.
  4. (Strongly recommended) open MATLAB, change directory to root directory of the program, run hack_javapath.m, and restart MATLAB (must restart MATLAB for the change to take effect).
    • This is needed because the program depends on some Java packages yet some of them are already shipped with the MATLAB, but in older versions, which will shadow the packages provided by cdttable. This hack_javapath.m will make the packages from my program take precedence. The old packages from MATLAB are too old so that they will usually break cdttable. Of three versions I tested, only R2015b works without this hack. Notice that this hack would affect MATLAB globally, and you may want to revoke it when not using the package.
    • to revoke the hack, just replace the result of evaluating fullfile(matlabPrefDir, 'javaclasspath.txt.bak') with fullfile(matlabPrefDir, 'javaclasspath.txt') if the first file exists. Otherwise simply delete the second one.
  5. Every time before using the program, do the following.
    • open MATLAB and change directory to root directory of cdttable.
    • run initialize_path in MATLAB to add path. Or you can do this manually.

Third party libraries

This lists all third party dependencies included in cdttable.

Run the demo program

change directory to /demos and run import_NEV_demo. You should see a file import_NEV_demo_result.mat generated under /demos/import_NEV_demo_results.

Todo

add link to template specification, parameter specification.

What’s done by the demo

Basically, this demo program converts two NEV files into two CDT tables, one for each file. These two NEV files are the part of the data for an old experiment in our lab. During each trial, 3 stimuli were presented to the monkey in sequence, and there are 416 trials and 416*3 stimuli in total for each file (so each stimulus only presented once per file).

function import_NEV_demo()
% IMPORT_NEV_DEMO ...
%
%   a demo showing importing NEV files using +nevreader and +cdttable package.
%
% Yimeng Zhang
% Computer Science Department, Carnegie Mellon University
% zym1010@gmail.com

%% DATE      : 23-Jul-2015 16:45:15 $
%% DEVELOPED : 8.3.0.532 (R2014a)
%% FILENAME  : import_NEV_demo.m

import cdttable.import_files


%% load file list.
basePathNEV = fullfile(root_dir(),'demos','import_NEV_demo_data');
basePathCTX = fullfile(root_dir(),'demos','import_NEV_demo_data');

NEV_list = {
    fullfile(basePathNEV,'v1_2012_1105_003.nev');
    fullfile(basePathNEV,'v1_2012_1105_004.nev');
    };

CTX_list = {
    fullfile(basePathCTX,'GA110512.3');
    fullfile(basePathCTX,'GA110512.4');
    };

%% get path for parameters
templatePath = fullfile(nevreader.root_dir(), 'demo_trial_templates', '3ec_edge_or.json');
preprocessParamsPath = fullfile(nevreader.root_dir(), 'demo_preprocessing_params', 'default.json');
importParamsPath = fullfile(cdttable.root_dir(), 'demo_import_params', 'edge_test.json');

argList = cell(numel(NEV_list),1);

for iFile = 1:numel(argList)
    argList{iFile} = cell(4,1);
    argList{iFile}{1} = NEV_list{iFile};
    argList{iFile}{2} = preprocessParamsPath;
    argList{iFile}{3} = templatePath;
    argList{iFile}{4} = CTX_list{iFile};
end

%% do the actual conversion.
[CDTTables, import_params_string] = ...
    import_files(@nevreader.preprocessing_func, argList, importParamsPath);



%% save the result.
saveDir = fullfile(root_dir(),'demos','import_NEV_demo_results');
saveFileName = 'import_NEV_demo_result'; % no extension, which will be appended automatically.
% a custom meta field. set metaInfo = [] or just don't pass it into
% save_CDTTable if you don't need it.
metaInfo = struct();
metaInfo.NEV_list = NEV_list;
metaInfo.CTX_list = CTX_list;
tablename_list = NEV_list;
metaInfo.tablename_list = tablename_list;

metaInfo.timestamp = datestr(now,30);
metaInfo.import_params_string = import_params_string;

save_CDTTable(fullfile(saveDir,saveFileName),CDTTables,metaInfo);

end

function save_CDTTable(savePath, CDTTables, metaInfo)
% demo function to save result. maybe it's good to change it into a
% standalone function.

if nargin < 3 || isempty(metaInfo)
    metaInfo = [];
end

save([savePath '.mat'],'CDTTables','savePath','metaInfo');


end

  1. In the first cell block of the code (%% load file list.), the filelists of NEV and CTX files are constructed. CTX files are just to double check the correctness of event codes in the NEV files and can be omitted.

  2. In the second block (%% get path for parameters), the paths of parameter files for the conversion are specified. There are three parameter file, two (templatePath, preprocessParamsPath) for the +nevreader package to convert NEV to a common intermediate format, which is specified in Common Intermediate Format (CIF), and the third one importParamsPath specifies how to extract and align event codes. The first two files are specific to +nevreader and will be ignored for now, except that you should know they convert NEV files into the Common Intermediate Format (CIF). The content of the third file (which is a JSON document) is as follows.

    {
  "comment":"params for experiment edge_test, in Corentin's data",
  "subtrials":[
    {
      "start_code":25,
      "end_code":26
    },
    {
      "start_code":27,
      "end_code":28
    },
    {
      "start_code":29,
      "end_code":30
    }
  ],
  "trial_to_condition_func":"(NEV_code, idx) (NEV_code(6)-192)*64+(NEV_code(7)-192)+1",
  "margin_before":0.3,
  "margin_after":0.3
}
    • comment is just some description of the import parameters, and is ignored by the program.
    • subtrials is the most important field in the file, whose elements specify the markers of three stimuli in each trial. In this case, stimulus 1 ON/OFF is marked by 25/26, stimulus 2 ON/OFF by 27/28, and stimulus 3 ON/OFF by 29/30.
    • trial_to_condition_func specifies how to extract the condition number from event codes in the trial and the trial index. In this case, the condition number is encoded by the quotient and and remainder of the condition number divided by 64, and the quotient and remainder are added by 192 in the end to avoid confusion with other event codes.
    • margin_before and margin_after specifies the margins in spike extraction window. Given the three subtrials as specified above, for each trial, the program would collect all spikes and events between (t(25) - margin_before, t(30) + margin_after), where t(x) refers to the timestamp for event code with value x. For example, if for some particular trial, code 25 appears in 133.5 second (relative to some reference time, say start of recording), and code 30 appears in 134.0 second, then the program would collect all codes and spikes during time window (133.2s, 134.3s) (notice it’s open interval; it’s for compatibility with legacy program in our lab). For more info on how to specify the import parameter file, see Import Parameters File Specification.

  3. In the third block (%% do the actual conversion.), the actual conversion is performed, by calling +cdttable.import_files(), and passing in the preprocessing function for NEV files +nevreader.preprocessing_func(), the arguments to pass into the preprocessing function argList, and the path of import parameter file importParamsPath.

  4. In the fourth block (%% save the result.), the resultant CDT tables are saved as import_NEV_demo_result.mat.