Doco is available on the SAS Support site here:

http://support.sas.com/documentation/solutions/performmanagement/stm/index.html

You will need to email your friendly SAS Support team for a password (only if you have a SPM license though)

The reason I wanted to know is that over time there seems to be records in there and the table seems to just keep growing, especially in our development environment when we were doing lots of test SPM loads and calcs. When you are adding 10,000 odd rows to SPM every week this table had me almost as concerned as the sas_elementcolumn table.

A track to SAS Tech Support on this came back with nothing should be in there, truncate it.

We are just finishing moving all our SPM loads from our tried and true ‘out-of-the-box’ SAS DI Studio jobs (which no longer works with the latest SAS 9.1.3 Foundation patch), to the new ‘build it yourself’ BMF loads (please don’t ask!)

So I was watching the Mysql database today to see if I could diagnose what was causing a performance issue with the calcs, and finally I think I undertsand one of the uses for the sas_wdetemptable.

So this is what I think happens…..

When you submit the calc, SPM:

1) Reads any rows in the sas_elementcolumn table for the project/period you are calcing for (so in our case 120,000 reads).

2) Writes these records to the sas_wdetemptable (120,000 writes)

3) Deletes the records from the sas_wdetemptable (120,000 deletes)

4) Somehow merges the records in the sas_wdetemptable with the new calc’d value (no idea on this one as I couldn’t see it ) and then inserts them back into the sas_elementcolumn table (120,000 inserts)

5) Deletes these records from the sas_wdetemptable (120,000 writes)

So now I know why the calc takes 40 minutes and the Mysql database seems to get fragmented.

And the reason we had some left, I think it is because every now and again the Web Server or Mysql instance would get stopped mid calc in Dev, and then the records in sas_wdetemptables won’t get deleted (its got me think about whats in, or more importantly not in sas_elementcolumns though when this happens)

Weird thing is the Mysql database is not using a lot of memory or cpu for this work, so need to read up on tuning this beast me thinks.

By default MySql creates a single file on the server for the database. Not bad if you are on Unix but if you are on windows (and with SPM you are probably on Windows) then when SPM gets large the single database file gets large.

Having a single file on windows that is over 2GB is a bad idea (in my opinion anyway and especially if you are on 32 bit windows and of course i you are on SPM 2.x you are on 32 bit windows)

One of the other challenges with MySQL is unlike some other databases it just seems to grow and grow and never reclaims any of its internal space. In fact the only way to reclaim space ans shrink the database file is to export the entire database, delete the database, recreate it and reimport the exported file. Time consuming and fraught with risk.

However there is an answer…. (well a suggestion anyway)

You can tell MySql to create a number of smaller files to effectively partition the database files. How I hear you ask…..

Well in the my.ini for the database added these lines:

innodb_data_home_dir=drive:/dir1/dir2/dir3/ibdata

innodb_data_file_path=ibdata1:2000M;ibdata2:2000M;ibdata3:2000M;ibdata4:2000M;ibdata5:2000M;ibdata6:2000M;ibdata7:2000M;ibdata8:2000M;ibdata9:2000M;ibdata10:2000M;ibdata11:2000M;ibdata12:2000M;ibdata13:2000M;ibdata14:2000M;ibdata15:2000M;ibdata16:2000M:autoextend

innodb_log_group_home_dir=drive:/dir1/dir2/dir3/iblogs

So the first line says where tostore the db files.

The second line says to create 16 2GB files and then automatically extend the last one to infinity (and beyond if the MySql database needs to go over 32 GB.

One downside is that MySql will create all the files when it starts up, so you will suddenyl have consumed 32GB of disk. You might want to tune the number of files based on your environment.

Lastly before you do this you will need to backup your MySql instance, stop the Mysql service, remove the old MySql DB files and then restart MySql, which will recreate the DB files with the new partitonedversons automagically.

Then reimport the backup and wahoo you are away (plus ypu have just regorg’d the db and reclaim any lost space as well).

And of course you will ask, yes we asked SAS Tech Support and yes they ‘indicated’ this was not unsupported whichmeans it is supported.

Partition away!

"SAS Software Documentation 

Documentation for this product is available for customers only. Please contact SAS for details about accessing this documentation.

See your product HELP system for additional information. "

Over on the SAS Support site.

http://support.sas.com/documentation/onlinedoc/spm/

(thanks to the friendly Presales dudes for the pointer)

Over on the Microsoft BI Blog there is a post lauding how they have rave reviews for DATAllegro warehouses with data volumes over 200+ TB and amasing query response times.

So what will the move to Widows and SQL Server bring?

What will the migration path be like?

How will customers feel about the requirement to constantly apply windows patches to their “appliance”?

How many 200TB SQL Server/Windowswrehouse sites are out there at the moment?

Given the move from one technology stack to another is always fraught with difficult (just think SAS FM, SPM and ABM) then I would hate to be a DATAllegro customer with a 200TB warehouse at the moment.