You are here: Setup > Marketplace Tools > Directory Walker

Directory Walker

The Directory Walker is a Catalogic utility that can be used in conjunction with ECX. It was developed based on customer demand where there was a need to understand disk consumption originating from NetApp storage on a customer basis. The customer requirement was a report that presents a NetApp host, the path corresponding to a volume that is specific to the end user, total storage consumption of that path, and file/directory count for the path. This should be generated in a recursive manner such that detailed information can be provided at a sub-path/sub-directory level.

The Directory Walker is a self-executable jar file that simply executes ECX API queries against the NetApp cataloged metadata, aggregates the query results, and then logs the statistics to one or more files. The app supports setting options in multiple forms. Options can be submitted via the command line, properties file or via an optional HTML 5 GUI.

Required Components

NetApp Storage System(s): One or more NetApp storage systems that meet the requirements as detailed in the System Requirements.
Client Machine: A client machine with a browser installed to access the ECX portal.
ECX: ECX, installed as a virtual appliance. Required resources have to be added.

Key Steps

Install Directory Walker
Run a Catalog Data Policy
Run a Directory Walker Policy using the script policy feature of ECX
Gather output files

Install Directory Walker

Select Directory Walker in the Catalogic Marketplace, then click Install.

Run a Catalog Data Policy

Create a Catalog Data Policy

A Catalog Data policy provides the framework to collect and catalog information about objects on a registered resource. In the ECX portal, click the Plan tab and the Policies view. Click New and then select a catalog data policy. Select one or more resources to catalog from the list of available resources. Select the options for your policy. Also, enter notifications. If notification options are enabled, an email message with information about the status of each task is sent when the job completes. Optionally, select one or more defined schedules for your policy and save the policy as a job.

Run a Job

A job that is based on a Catalog Data policy discovers object information, catalogs it, and populates the ECX database. Click the Monitor tab and the Jobs view. Select the job to run by clicking in the row containing the job name. Click Start. The job session runs.

Run a Directory Walker Policy

Create a Script Policy

In the ECX portal, click the Plan tab. On the Views pane, select Policies.

Click the All Policies tab, click New, then click Script. The Script Policy Editor opens. Click the 1: Scripts tab. On the Scripts pane, enter the Directory Walker command in the command text box, and optionally enter a tag or remark in the corresponding description text box. Then click Add.

The next section describes the syntax and options for the Directory Walker command.

Run a Job

Click the Monitor tab and the Jobs view. Select the job to run by clicking in the row containing the job name. Click Start. The job session runs.

Directory Walker script policy options

All options, except for logsuffix and logpath, can be submitted via the command line or properties file. Note that options included on the command line will overwrite option settings in the configuration file. Following are the supported options:

Directory Walker script policy options
Options	Description
Dlogsuffix	Allows the name of logs to be dynamically modified
Dlogpath	Allows the location of logs to be dynamically modified
Debug Mode	Allows for logs to include debug level statements
Help
Detail Logging	Produces a data set (dw_detail.log) containing directory path details with the following format: <date time>,<full path>,<directory size in bytes>, <object count> The data set is limited to no more than 100 files at 10MB each.
Directory Depth	Determines the path depth that will be recorded in the dw_detail.log file
ECX User	Corresponds to the ECX appliance user account
ECX Password	Corresponds to the ECX appliance’s user password
NetApp Controller Name	Specifies the URL of the appliance where ECX is running. If run within the ECX container, this value is not needed.
Page Size	Determines how many records will be requested from ECX and processed
Volume Name	Determines which volume to be processed
Volume Path	Determines the path(s) to be processed

Examples of the Directory Walker command with command line options:

/usr/java/latest/bin/java -Dlogsuffix="DW" -jar /opt/ECX/tools/scripts/directorywalker_023.jar -l -depth -1 -ecxuser <username> -ecxpass <password> -host <hostname> -pageSize 3000 -volumeName <volumename> -path </XXX,/YYY,/ZZZ>

Examples of the Directory Walker command with options in a configuration file:

/usr/java/latest/bin/java -Dlogsuffix="DW"

-DECX_PROPS=/opt/ECX/tools/scripts/application_dw.properties -jar /opt/ECX/tools/scripts/directorywalker_023.jar

The above should be entered as a single line in the script policy editor.

The Directory Walker configuration file

#################### ECX USER ######################

# Must be a valid user configured within ECX

####################################################

catalogic.ecxuser= <username>

#################### ECX Password######################

# Must be a valid user's password configured within ECX

####################################################

catalogic.ecxpass= <password>

################# ECX URL ######################

# Must be a valid URL to ECX. If the jar is being execute

# within the ECX appliance then the http://localhost:8082 is fine

# if the utility is run outside the appliance then the URL

# should include a host name or IP other than localhost.

####################################################

catalogic.serverURL= http://xxx.xxx.xxx.xxx:8082

################# host ######################

# FAS controller name that has been cataloged by ECX

####################################################

catalogic.host= <hostname>

################# volumeName ######################

# the volume name of the FAS device where the path will be found

####################################################

catalogic.volumeName= <volumename>

################# path ######################

# path you are interested in rolling up. All paths start with /

####################################################

catalogic.path = /XXXX,/YYYY,/ZZZZ

################# logDetail ######################

# true/false if the value is true then dw_detail.log will be created.

# A new set of files will be created with each execution. The configuration

# will prevent the log file set from exceeding 1 gig.

####################################################

catalogic.logDetail = true

################# pageSize ######################

# the amount of records to process at a time. The higher this value is set the

# faster the tool will process. However, the higher the

# the page size the more memory is required for processing

####################################################

catalogic.pageSize = 5000

################# directory depth ######################

# this just controls how much data is printed to the

# detailed log when catalogic.logDetail = true; When set to -1,

# all directory and subdirectory paths are printed to the detail log

# which match the path criteria. If the value is 3 for example,

# the detailed log will record all paths that are no longer than 3 levels deep.

####################################################

catalogic.depth = -1

################# error retry ######################

# this just controls how many attempts upon failure a query against ECX will be attempted.

####################################################

catalogic.errorRetry = 5

Gather output files

By default the output files are stored in /opt/ECX/virgo unless differently specified with the Dlogpath option. Output files consist of dw_main.log and dw_details.log. The name is dependent on the Dlogsuffix option.