NSEIndia Data Downloader
- Thread starter prestonmyers2
- Start date
Hello Murthy,
I corrected those errors, The dates are like that because when 'USe Date as name' option is seklected I thought the file name should have the start and end dates in them!!
.
Regards,
Preston.
I corrected those errors, The dates are like that because when 'USe Date as name' option is seklected I thought the file name should have the start and end dates in them!!
.Regards,
Preston.
Checked, Ok.
Your historical file name format TickerFromdateTodate is also logical.
Thanks, My next report will be long and after a longtime, matching with your return, in which all small issues can be addressed.
murthymsr
Hi preston,
I felt that I present a list of requests, that will make the program simpler and more functional. You may have a look and incorporate as feasible.
I have been regularly using NSEDD for inputting data into AmiBroker and it is working flawlessly.
I have for testing purpose, tried with FCharts and the data is compatible. You might have seen the screen shot posted by me earlier.
I will install FiboTrader and other popular free TA software and will test with NSEDD data after the changes proposed now, as some of the requests will have bearing on data suitability.
REMOVABLE OPTIONS
1) Portable Feature : We may make the program as a portable application, to facilitate running the program from pen drive and moving the program and data across different systems etc. As no registry entries are added / modified, this can be easily be done.
Presently, there is a facility by which user selects the folder into which he will download EQBE / Historical / Live data. Instead, the program may be delivered with EMPTY FIXED sub-folders and the downloaded/ converted data is saved into those respective folders ONLY. In addition to providing portability, this will greatly simplify the user interface.
But the folder opening option presently available is a cool one and may be required. To achieve this, we may have a single 'folder open' button to the right of progress box, on clicking of which, it will open the folder into which the last download operation took place.
2) General Options : Remember Last Date
This feature when checked is very useful and should always be in checked mode. i.e., may be removed and the logic built in accordingly.
After the user downloads and establishes the database for the desired few months or years, he will be updating the data every day, either at the evening of the trading day or the morning of the next trading day. So, the program may remember the last 'To Date' add 1 and make it the 'From date' for the next run. The 'To Date' will be the current day. With this logic, in both cases, the date range remains logical and the check box is unnecessary. Hope my view is clear.
3) BhavCopy Options : Date Format
This option will presently converts the date field of the converted data and also the dates when displayed in the filename. FCharts will easily identify the date format YYYYMMDD, which is unambiguous. If the date format is anything other than YYYYMMDD, it will interpret the date with reference to the date format settings in the windows international settings. As the date, contained within the data is meant for the charting software only ad the user does not normally see it, the date format may be YYYYMMDD always. This does not pose a problem with other TA software also.
This option may be removed altogether and the file names may be in YYYYMMDD format to facilitate easy sorted view to identify the missing dates and all other dates like from date and to date may be in the DDMMYYYY format, which the Indian users are accustomed to.
4) Bhavcopy Options : Add Header
This option also is NOT necessary. Instead, the program may ALWAYS add the header. In AB, within import settings file, this can be set. In FCharts it automatically recognizes the header when present. With Fibotrader and Metastock, the header is a MUST. By always adding the header, we will get data compatibility with Fibotrader and Metastock also.
Presently, the header inserted is:
Ticker,Date,Open,High,Low,Close,Vol
Instead, the header line may be: (Insert each field Name within angle brackets)
Ticker,Date,Open,High,Low,Close,Vol
<Ticker>,<Date>,<Open>,<High>,<Low>,<Close>,<Volume>
This change will make it Fibotrader and Metastock compatible without losing the AB & FC compatibility. Now different files are using different formats. All may use this format for columns, and their header always.
This brings the data to metastock data format standard. Metastock is THE most popular data standard and any Charting software will surely make his program accept metastock data format.
OPTIONS FOR DEVELOPER & ADVANCED USER:
5) General Options : File Extension Text box
Default extension may be TXT. This option may be removed from the user interface, but still be available for changing from the INI file.
6) BhavCopy Options : Change Format
Now this option is available to user for manipulation. In my view this option may be required for the developerand the advanced user ONLY and is NOT required for the normal regular user. So, again, in my view this option may be removed from the Bhav copy options tab. This option, however, IS available in the options.ini file and can be used when necessary.
7) Bhav copy Options : Use Default Names
This also may be removed from the user interface but still will be available from the INI file, for reasons sama as for 'Change Format'.
8) Bhavcopy Options : Remove spaces
This also may be removed from the user interface but still will be available from the INI file, for reasons sama as for 'Change Format'. Moreover, if user makes a mistake, the downloaded data will be useless for importing into charting software.
OTHER REQUESTS
9) Historical Options : Historical Indice and the Historical Yield
These are not the program options, but are the user selections for the services. So, I feel, that these may be brought out within the Historical Data module like EQBE and FO in the BhavCopy module.
10) Facility to rename indice: Some users may use yahoo for intra-day data. Others may use NSE (through QT or other sources). Different sources use different names to the same index. To facilitate proper integration and combined look-up of EOD and Intra-day quotes, Indice may be allowed to be renamed as per user's choice, through options and may be saved to INI file.
11) The Indice data downloaded does not have the ticker column. This also may be provided, even if the information is available in the file name. Same is the case with yield.
12) Feature which saves the program window size and position into the INI file and restoring through the next start-up. This will help users to Use different applications simultaneously and with each application positioning in it's assigned location for ease of combined use.
13) The application window size is presently about height= 568 and width = 651. The height may be fixed to a max of 540 and the width may be set to a max of 720. These limits will help users with different display resolutions from 800 X 600 (old machines) onwards to at least 1280 X 800.
Thanks.
murthymsr
I felt that I present a list of requests, that will make the program simpler and more functional. You may have a look and incorporate as feasible.
I have been regularly using NSEDD for inputting data into AmiBroker and it is working flawlessly.
I have for testing purpose, tried with FCharts and the data is compatible. You might have seen the screen shot posted by me earlier.
I will install FiboTrader and other popular free TA software and will test with NSEDD data after the changes proposed now, as some of the requests will have bearing on data suitability.
REMOVABLE OPTIONS
1) Portable Feature : We may make the program as a portable application, to facilitate running the program from pen drive and moving the program and data across different systems etc. As no registry entries are added / modified, this can be easily be done.
Presently, there is a facility by which user selects the folder into which he will download EQBE / Historical / Live data. Instead, the program may be delivered with EMPTY FIXED sub-folders and the downloaded/ converted data is saved into those respective folders ONLY. In addition to providing portability, this will greatly simplify the user interface.
But the folder opening option presently available is a cool one and may be required. To achieve this, we may have a single 'folder open' button to the right of progress box, on clicking of which, it will open the folder into which the last download operation took place.
2) General Options : Remember Last Date
This feature when checked is very useful and should always be in checked mode. i.e., may be removed and the logic built in accordingly.
After the user downloads and establishes the database for the desired few months or years, he will be updating the data every day, either at the evening of the trading day or the morning of the next trading day. So, the program may remember the last 'To Date' add 1 and make it the 'From date' for the next run. The 'To Date' will be the current day. With this logic, in both cases, the date range remains logical and the check box is unnecessary. Hope my view is clear.
3) BhavCopy Options : Date Format
This option will presently converts the date field of the converted data and also the dates when displayed in the filename. FCharts will easily identify the date format YYYYMMDD, which is unambiguous. If the date format is anything other than YYYYMMDD, it will interpret the date with reference to the date format settings in the windows international settings. As the date, contained within the data is meant for the charting software only ad the user does not normally see it, the date format may be YYYYMMDD always. This does not pose a problem with other TA software also.
This option may be removed altogether and the file names may be in YYYYMMDD format to facilitate easy sorted view to identify the missing dates and all other dates like from date and to date may be in the DDMMYYYY format, which the Indian users are accustomed to.
4) Bhavcopy Options : Add Header
This option also is NOT necessary. Instead, the program may ALWAYS add the header. In AB, within import settings file, this can be set. In FCharts it automatically recognizes the header when present. With Fibotrader and Metastock, the header is a MUST. By always adding the header, we will get data compatibility with Fibotrader and Metastock also.
Presently, the header inserted is:
Ticker,Date,Open,High,Low,Close,Vol
Instead, the header line may be: (Insert each field Name within angle brackets)
Ticker,Date,Open,High,Low,Close,Vol
<Ticker>,<Date>,<Open>,<High>,<Low>,<Close>,<Volume>
This change will make it Fibotrader and Metastock compatible without losing the AB & FC compatibility. Now different files are using different formats. All may use this format for columns, and their header always.
This brings the data to metastock data format standard. Metastock is THE most popular data standard and any Charting software will surely make his program accept metastock data format.
OPTIONS FOR DEVELOPER & ADVANCED USER:
5) General Options : File Extension Text box
Default extension may be TXT. This option may be removed from the user interface, but still be available for changing from the INI file.
6) BhavCopy Options : Change Format
Now this option is available to user for manipulation. In my view this option may be required for the developerand the advanced user ONLY and is NOT required for the normal regular user. So, again, in my view this option may be removed from the Bhav copy options tab. This option, however, IS available in the options.ini file and can be used when necessary.
7) Bhav copy Options : Use Default Names
This also may be removed from the user interface but still will be available from the INI file, for reasons sama as for 'Change Format'.
8) Bhavcopy Options : Remove spaces
This also may be removed from the user interface but still will be available from the INI file, for reasons sama as for 'Change Format'. Moreover, if user makes a mistake, the downloaded data will be useless for importing into charting software.
OTHER REQUESTS
9) Historical Options : Historical Indice and the Historical Yield
These are not the program options, but are the user selections for the services. So, I feel, that these may be brought out within the Historical Data module like EQBE and FO in the BhavCopy module.
10) Facility to rename indice: Some users may use yahoo for intra-day data. Others may use NSE (through QT or other sources). Different sources use different names to the same index. To facilitate proper integration and combined look-up of EOD and Intra-day quotes, Indice may be allowed to be renamed as per user's choice, through options and may be saved to INI file.
11) The Indice data downloaded does not have the ticker column. This also may be provided, even if the information is available in the file name. Same is the case with yield.
12) Feature which saves the program window size and position into the INI file and restoring through the next start-up. This will help users to Use different applications simultaneously and with each application positioning in it's assigned location for ease of combined use.
13) The application window size is presently about height= 568 and width = 651. The height may be fixed to a max of 540 and the width may be set to a max of 720. These limits will help users with different display resolutions from 800 X 600 (old machines) onwards to at least 1280 X 800.
Thanks.
murthymsr
Hello Murthy,
Thanks for the input I appreciate it. I am at work currently and cannot find time to modify the program but when I get home I will surely look at your points and try to incorporate them.
Thanks & Regards,
Preston.
Thanks for the input I appreciate it. I am at work currently and cannot find time to modify the program but when I get home I will surely look at your points and try to incorporate them
.Thanks & Regards,
Preston.
murthymsr
Dear Preston,
Excellent utility.
The EOD export is really flawless.
I have seen the Live Quotes Section in the 1st December 2008 update but was unable to figure out how to import intraday data into the Amibroker database.
Would appreciate if you could brief me as to how to to do that..
Regards
Anshuman
Excellent utility.
The EOD export is really flawless.
I have seen the Live Quotes Section in the 1st December 2008 update but was unable to figure out how to import intraday data into the Amibroker database.
Would appreciate if you could brief me as to how to to do that..
Regards
Anshuman
Hi Preston,
Feature requests / Observations (continued from previous post)
14) Presently, the program is delivered with symbollist.CSV, containing the yahoo tickers for Live quotes, NSE tickers for Live quotes and also NSE historic tickers and also IndiceSymbolList.TXT in which the tickers are separated with commas.
I suggest that the program may be delivered with THREE separate files with TXT extension namely YahooLive, NSELive and NSEHistoric. The files may contain ONE ticker per row only.
When the second field is 'Y' or 'y',then only the data for that ticker is downloaded. This will help many users who will select from within a preset list.
Where an alias name is required by user like in yahoo live indice / stocks and NSE Indice, he may indicate the same in the next (3rd) field). the three fields are comma separated.
The NSEHistoric file may contain all the indice also and no separate file is required. While downloading the EOD data and when indice is included, the alias names are used in renaming the tickers. This will greatly simplify the Histocical data segment with a combo box for indice, three click buttons and a file open button.
15) Yahoo Live quotes fetches and saves to CSV file the fields Ticker, date, time, %change, open, high, low, close, volume without header line. I feel that %change,open,high and low fields are not necessary for Intraday TA. Even the date may be fetched from system date. It may be sufficient to fetch only time,close(last) and volume. may be sufficient. This will save lot of bandwidth. The file may be saved asa TXT file. The volume presented will be incremental volume and the first record may be deleted after incremental volume calculation as the first incremental volume will be the volumefor that day UPTO THAT MOMENT.
16) NSELive quotes fetches and saves to CSV file the fields Symbol, date,time,open,high,low,last, previousClose, chvalue, chpercent, avgprice, tradedqty, turnoverlakhs without any processing. The data may be processed and presented in the standard metastock data format with header. Then only this file will be able to be used by the TA software.
17)The historical yield option bears no relavance as far as data for TA software is concerned. This option may be removed. If You intended to do TA on this, you may present this data in the standard ticker,date,OHLCV fields. PERatio and PBRatio will be two different 'tickers' with all OHLC taking the same value and V is zero like in index data.
18) The Index file contains the fields date and OHLC. This also may be in the standard format ticker,date, O, H, L, C and volume. (which is zero for NSE indice)
19) The Historical download for ticker data is presented in the raw form. This may also be presented in the standard
format for TA.
20) File naming conventions: All the files may be named with the same naming convention in their respective folders.
EOD files for a day may be named like YYYYMMDD and TXT extention.
EOD files for a date range may be named like FromDateInYYYYMMDDFformat + "_" + ToDateInYYYYMMDDFformat with TXT extention.
Stock and Indice Historical files may be named as stockname +"_"+ date range as above.
21) In Historical file options, we have check boxes for 'Historical Indice' and 'Historical yield'. I unchecked both and pressed the 'Ticker History' button. Stock Historical data is downloaded. I think that 'Historical Indice' checkbox is not necessary and the 'Historical yield button' may be renamed as 'Add Historical Yield.
Feature requests / Observations (continued from previous post)
14) Presently, the program is delivered with symbollist.CSV, containing the yahoo tickers for Live quotes, NSE tickers for Live quotes and also NSE historic tickers and also IndiceSymbolList.TXT in which the tickers are separated with commas.
I suggest that the program may be delivered with THREE separate files with TXT extension namely YahooLive, NSELive and NSEHistoric. The files may contain ONE ticker per row only.
When the second field is 'Y' or 'y',then only the data for that ticker is downloaded. This will help many users who will select from within a preset list.
Where an alias name is required by user like in yahoo live indice / stocks and NSE Indice, he may indicate the same in the next (3rd) field). the three fields are comma separated.
The NSEHistoric file may contain all the indice also and no separate file is required. While downloading the EOD data and when indice is included, the alias names are used in renaming the tickers. This will greatly simplify the Histocical data segment with a combo box for indice, three click buttons and a file open button.
15) Yahoo Live quotes fetches and saves to CSV file the fields Ticker, date, time, %change, open, high, low, close, volume without header line. I feel that %change,open,high and low fields are not necessary for Intraday TA. Even the date may be fetched from system date. It may be sufficient to fetch only time,close(last) and volume. may be sufficient. This will save lot of bandwidth. The file may be saved asa TXT file. The volume presented will be incremental volume and the first record may be deleted after incremental volume calculation as the first incremental volume will be the volumefor that day UPTO THAT MOMENT.
16) NSELive quotes fetches and saves to CSV file the fields Symbol, date,time,open,high,low,last, previousClose, chvalue, chpercent, avgprice, tradedqty, turnoverlakhs without any processing. The data may be processed and presented in the standard metastock data format with header. Then only this file will be able to be used by the TA software.
17)The historical yield option bears no relavance as far as data for TA software is concerned. This option may be removed. If You intended to do TA on this, you may present this data in the standard ticker,date,OHLCV fields. PERatio and PBRatio will be two different 'tickers' with all OHLC taking the same value and V is zero like in index data.
18) The Index file contains the fields date and OHLC. This also may be in the standard format ticker,date, O, H, L, C and volume. (which is zero for NSE indice)
19) The Historical download for ticker data is presented in the raw form. This may also be presented in the standard
format for TA.
20) File naming conventions: All the files may be named with the same naming convention in their respective folders.
EOD files for a day may be named like YYYYMMDD and TXT extention.
EOD files for a date range may be named like FromDateInYYYYMMDDFformat + "_" + ToDateInYYYYMMDDFformat with TXT extention.
Stock and Indice Historical files may be named as stockname +"_"+ date range as above.
21) In Historical file options, we have check boxes for 'Historical Indice' and 'Historical yield'. I unchecked both and pressed the 'Ticker History' button. Stock Historical data is downloaded. I think that 'Historical Indice' checkbox is not necessary and the 'Historical yield button' may be renamed as 'Add Historical Yield.
Similar threads
-
-
-
-
-
Intraday Options data from nseindia.com to EXCELL
- Started by ranger123
- Replies: 1