h1. DQO TODO h3. IDL * Check out newly changed code in the dq repository h3. Transfer everything from here to Teambox h3. Web Dev - DQHANDS * TeamBox ** Documentation Software ** Project Management Software ** Bug Tracking Software * Directory Structure was created because of the server setting of the "cgi-bin" folder access. If it can be changed to anything, then the the web framework can be restructured to have semantic meaning * URL Path redirection after request submission * UX ** set up ui testing with gorilla testing ** Ask how do they use the web app * change views to use method calls * utilize rgba for css shadowing * move the newest versions on jquery into app js directory * LAYOUT ** Add Contact page (discuss) * REQUEST Page ** help dialog ** Big request might take a long time to load *** use a loading sequence before refresh * DATA DISPLAY Page ** IE8 not working ** get more request AJAX using post method ** created json data needs to use arrays instead of key/value pairing to get correct ordering display ** hour/field label highlight on hour cell hover ** Testing with svn ** check svn ** Diagnostic Plots *** Add loader gif *** link click calls a cgi-script to generate the page *** (KEN) Diagnostic plot directory doesn't match the json metric files *** same window: use global variable in js to store the window object and reuse by utilizing URL change to retrieve new plots *** environment variable to diagnostic plots: $DQPLOT_DATA **** window.open(url) **** stop at the date directory **** test: /home/kehoe/data/www/tool/dq ***** uri: http://dev.arm.gov/~kehoe/data/tool/dq/ *** start at pulling all plots from the directory *** then pick one case to build the configuration to generalize for the other cases *** conf for comparison plots ** ncvweb Plot *** href="http://plot.dmf.arm.gov/ncvweb/ncvweb.cgi?dd=1&page=plot&site=sgp&datastream=sgpmetE4.b1&file=sgpmetE4.b1.20101001.000000.cdf" target="new" ** Sample block coloring (discuss) *** concept 1 **** Original coloring implementation ***** Orders of importance: Bad(Red) > Indeterminate(Yellow) > Missing(Gray) > N/A(Black). ***** Green- only for 100% Good assessment. ***** If Good is < 100%, sample color will be based on the order of importance. Assessment of the highest order with percentage > 0% will color the sample block. ***** Mouseover will show percentage of colored assessment on the sample block (plus the table of assessment statistics; I will show the mock-up for it when I come in the next time). ***** Ex. Bad = 2%; Missing = 50% => block color: Red ***** Ex. Indeterminate = 40%; N/A = 40% => block color: Yellow **** Implement Assessment Rankings, but follow original coloring implementation ***** Default: the order listed in the json file *** concept 2 **** GOOD assessment: green, regularly **** SINGLE non-good assessment: color block will take that color **** MULTIPLE non-good assessment: color block will have a unified, standardized color for multiple non-good assessments **** toggling: ***** options listed will be the different types of assessments ***** clicking on an option will display only the color of the picked assessment in the block-data where it exists *** back-end code to be able to add more assessments than "add", "indeterminate", etc. *** Change the colors in the legend *** sgpmplpolfsC1:20101112 **** 99% on good but 0 on the others ** DQA Reports *** What we need to do for the DQA button is create a new URL that contains information from the DQ HandS website and add one more thing, the ARM ID number. Everyone in ARM has one. You can determine yours by logging into the ARM People database. Go to the ARM home page and look for a "People" link at the top of the page. After searching for your name in the database (last name), click update. Then look in the URL. That number is your ARM ID. So, somehow we need to add that to the URL that loads the DQAs. We currently have a local SQL database. I think the database is overkill. I think a JSON file with key=value pairs will work well. (ie kehoe=5880) For example the GRW MET URL could look like this: http://www.db.arm.gov/DQA/dqaenter.php?person_id=5880&site=grw&inst=met&sd=201007280000&ed=201010122359&fac=all The things we will need to pull from your Python script are: site=grw inst=met (instrument) sd=201007280000 (startdate including time date=20100728 time = 0000) The time can be made up and always this number. ed=201010122359 (enddate including time date=2010101 time=22359) The time can be made up and always this number. The only thing we will need to ask for is the user name and a password. We can use one password for all users, or set up another key in the JSON file. The user name would then be used to get the ARM ID. Then we just build the URL string and open a new window. ** plots for individual fields ** assessment toggling *** table functionality *** single field functionality *** UI ** dynamic add request ** Logs for different sites *** TWP and AMF use a system called contact sheets. (AMF is a general term meaning ARM Mobile Facility. We add one each year. Currently there have been FKB, GRW, HFE, NIM) All we would need to do is provide a link to one website. Inside that website the user can select the facility and instrument. **** http://www.twppo.lanl.gov/ops/cs/review.a4d *** NSA has one web site (Currently. There is talk of updating this system.) **** Instrument Logs, PM and CM logs->http://agvik.gi.alaska.edu/fmi/iwp/res/iwp_home.html *** SGP has three different sites. Unfortunately the sites are restricted by IP address. I had them add the NWC to the list, but we are no longer able to access the sites. I'm not sure why. **** CM logs = https://www.ops.sgp.arm.gov/menus/cmreports.html **** Instrument Logs = https://www.ops.sgp.arm.gov/Instrument_logs/Bsrnilog/ILogBSRNC1.html **** PM logs = https://www.ops.sgp.arm.gov/menus/pmreports.htm *** All of the sites currently have a configuration file in the $DQCONF_DATA directory. They are under the name of nsa.conf, sgp.conf, ... You can take a look at them. They are in Perl syntax. In the past, we could add the facility and instrument name to the Instrument Logs URL and it would take us directly to the correct web site. Without access to the website now, I'm not exactly sure how it is working. This is something we need to discuss with the site scientists more. ** Legend *** dynamically add assessment class during processing *** Button to display and hide the legend *** Move assessment color legend into each box (HOLD) ** Bookmarking *** Figure out how to write to database *** cache the shortened link after the first time *** Have a feature (a button with a pop-up) for bookmarking. the shown string is the query to get the same current request. This will clean up the URL *** When the request is passed into the page, have that string injected into a non displayed element. Then the javascript will look for the id, then extract the string, remove the element out of the dom, and then parse it. *** Write my own URL shortener using CGI *** http://stackoverflow.com/questions/136458/how-do-i-with-javascript-change-the-url-in-the-browser-without-loading-the-new *** http://api.bit.ly/v3/shorten?login=trinonsense&apiKey=R_9ec570bc385916500a54394d7a2ae11c&format=json&longUrl=http://betaworks.com/ *** http://code.google.com/p/bitly-api/wiki/ApiDocumentation#/v3/shorten *** http://bit.ly/a/account ** display for vap inputs ** add more request div fixed on the left side of the page with tipsy information ** Have a input request parsing check to ensure request standard ** back to top functionality ** Help Dialog Box *** help button to upon a modal box for help with image previews ** hover over previous or next day with tool-tip of the dates ** remove 0% test/summary ** take off 'time' if there is no time ** REWRITE SUMMARY POPUP DISPLAY. Too long *** Make a dynamic popup javascript generator **** http://www.w3schools.com/tags/tag_td.asp *** http://dev.arm.gov/~noensie/dqhands_dev/cgi-bin/explorer?request=nsa.met.C1.20101003 ** Add field data on hover ** JSON Value error "time" and "percent" *** http://dev.arm.gov/~noensie/dqhands_dev/cgi-bin/explorer?request=sgp.met.E4.20101101 ** datastream info box (HOLD) ** ONLY "new request" will be in the dock (HOLD) ** Hover over displays percentage of assessment + row and column highlight ** Design not found data block (discuss) ** C_DATA_DISPLAY python script ** USE SCROLL LOAD INSTEAD OF AUTOMATIC LOAD ** Assessment Statistics Display ** REWRITE THE DISPLAY PAGE ** REALLY SLOW EVEN W/O PROCESSING *** Slow at string conversion to json python object @m_data_display *** initially load first ~15 requests. Ajax call for the rest of the requests- 5 requests per ajax call. *** at each request, inject a script tag that would make an ajax call for the next batch of request ** Fixed-sized table columns *** cutoff field names: faded at the end for fields *** cut-off fields will have a tooltip of the full name ** "add new request" add to the nav bar * REQUEST MENU PARTIAL ** Fix sorting in getFacilities.py ** one day option ** clean up selectors ** JS *** Submit request button action will be different for the different pages *** FIX: add request 11.07.10 - anything later (check request list) *** tooltip for undo button, remove +/- *** cache at each selection *** Update earliest date and last date upon facility selection change *** Restructure the source input (ex. site-datastream-facility) *** move the empty list notification logic into add/remove request functions *** "undo" effects timing issue with single remove *** "undo" loses event handlers *** Undo Stack *** remove submission confirmation (the html, css, & js is still around) ** CSS *** font sizing of date selection calendar ** using post method for the submit form to enable big request ** Use $DQHEALTH_DATA environment variable for the path to the metric directory (Replace TEMP_PATH in ajax python scripts) ** fix date naming: start/end -> first/last (HTML, variable names) ** fix getFirstLastDates: it doesn't get the last date correctly ** ajax call: change directory once structure is set ** Change request display. Single line ** Loading Sequence GIf during request selection ** confirm-submit and confirm-clear notification ** cache the ajax call during selection (CANCELLED) * Look over dq.arm.gov to mimic its design ** try to get a higher resolution of the logo * Research: caching ** browser caching ** proxy caching * gZippin * Check the whole project into svn * Environment of development and production (not just under my directory) * Change page title at each page * DATA_PATH needs to be centralized * New json metrics directory structure ** The data is located at /home/kehoe/data/health/dq/metric Under this directory you will find the directory tree we have talked about. Take a look and let me know what you think. After thinking about all the different datastreams that we will process, I've realized we need to have both the date and time in the JSON file name. 99% of the time the time will not matter and can be ignored, but when we look at Radiosonde files, there will be four distinct files per day that will need to be displayed independently. I think the new radar files will also have multiple files per day. My hope was that your code would be able to recognize the datastream and call a different display program for these special cases. We can talk more about this later. If you want to start writing your web program I suggest you create a similar directory tree and copy over the directories and files into your /home/noensie/data/health/dq/metric/ area. That way you can use the $DQHEALTH_DATA environment variable, which is what we plan to do. * footer dqo link to site * Create helper for HTML TAGS ** Use them in all controllers * Day summary?? h3. Web Dev - General * http://www.smashingmagazine.com/2008/11/13/15-useful-project-management-tools/ * http://drup.org/20-basecamp-alternatives-project-management * minimal idl doc * sqlite * automated NEW page generator script * cgi-framework: discard dynamic "content" insertion and use abstract dynamic container insertion h3. Miscellaneous * Learn Regular Expression * Create a tutorial page for using my IDL Modules * SWAWT * NetCDF 4 * NetCDF organization files using bits h3. IDL * Move IDL Documentation