Oracle’s Jump Into the Big Data Realm
Many of you may have seen that Oracle officially announced their new Big Data product offerings today. Included in that list is the Apache distribution of Hadoop, Oracle Loader for Hadoop, Oracle Data Integrator Adapter for Hadoop, the Oracle NoSQL Database, and Oracle's R Enterprise. On the Oracle big data pages it seems like there is some confusion as to whether R will run a custom Oracle version or an Open Source version. Knowing Oracle it may be both. They have been working on a version of R within the Oracle database and might be augmenting it with an open source version on the Hadoop appliance. Lets cover what is known so far about the different components.
Apache Distribution of Hadoop
I'm actually really surprised here. Larry Ellison has acquired a Mike Olson company previously (Sleepy Cat) and Mike now runs Cloudera. I would have bet good money that a partnership would have been formed. My guess is that Oracle looked at the management tools Cloudera provides and determined that it would be too hard to integrate into their Enterprise Manager product or that Cloudera's price was too high. They are leading the Hadoop support market right now and have a really good future. Hadoop comes with a very large open source ecosystem around it. This solution should be great for both Hadoop and Oracle.
Oracle Loader for Hadoop
From what I have heard this is a map reduce job that will format the resulting data set file into an Oracle Data Pump file to be loaded directly into the database. There has been the tools SQOOP to suck data out of the database, but now we have the other side of the coin. Hadoop is great, but joins are still a bit problematic and the BI tools around it don't match what can be done in the traditional database world. The loader should help companies figure out what the balance should be.
Oracle Data Integrator Adapter for Hadoop
The Data Integrator Adapter is similar to the Oracle Loader for Hadoop, but it extends Oracle's Data Integrator product to be able to execute and manage Hadoop jobs as part of an ETL process. It is well known that Hadoop can crunch and count numbers faster than the Oracle Database in many cases. This allows the ETL process to offload the heavy number crunching and then use the Loader to put the data into the Oracle database when complete.
Oracle NoSQL Database
For a long time it has seemed like Oracle was neglecting the Berkeley DB product and not making giant leaps forward. Berkeley has always been a fantastic product for key value stores. In fact many of the major key value stores today are underpinned with Berkeley. It looks like Oracle has updated the product to bring many of the missing distributed features into the new product. It will be interesting to see how the new Times Ten Database, NoSQL Database, Hadoop, and ExaData components work together in the Oracle BI tools over time.
Oracle R Enterprise
R has long been the language of choice for the statistics community. It isn't clear if Oracle will be using the open source R-project.org distribution or has released their own. My guess is both. R-project will be deployed with the Big Data appliance and Oracle's R within the database. This should make a bunch of the big data number crunchers from the SAS world happy.
I have been told that Larry and TK have given the green light to go full force into NoSQL. If that isn't justification that it's a "real thing" that is here to stay I don't know what is. Oracle has and will continue to invest significant resources into Big Data.
This makes the team here at UberEther happy to see the largest software vendor come to the table and support the Hadoop community. It relieves some of our worries in developing out new log aggregation and risk adjustable access control product to the market. We know our product will be able to run on predefined hardware platforms for our largest customers and we can easily load the data into their legacy tools to reuse their existing investments. We still have a lot of work to do but if you're interested in hearing some more while we're out here at Open World contact us and we'd be more than happy to show you what we've been working on.
Widget for Voting in the Mix!
So I've been missing from here, email, etc for a while but I'll address that later, for a sneak preview of why you can see my new disclaimer on the right hand side of this blog. If you've been paying attention to the guys over at AppsLab (and who wouldn't be), you'll have noticed that they added some new functionality to Oracle Mix to nominate and vote on sessions for this years Oracle Open World. Not too long after the launch Jake mentioned that they needed some new ideas on how to review the submitted sessions. I had an idea to create a wordpress widget that lists all the sessions in a random order. It took me a little while squeezing it out while life got in the way, but here it is:
If you want to check it out you can see it running in the sidebar of every page on this blog. It's a pretty generic plug-in, if you want to stream any feed from mix it should be pretty simple to modify, just remove the parsing for votes in the code. The installation is pretty simple:
Oracle Discoverer / BIP Integration Update
I keep seeing things pop up and people having trouble getting the Discoverer / BIP integration working. I will say this is not the easiest of configurations, but with some additional steps anyone should be able to complete the task. For those of you that have been following along, the latest Discoverer Cumulative Patch (CU4, p6357481) was released and the Discoverer / BIP integration was not included with created a new interop patch for everyone numbered 6622352.
Just a couple notes on the install. If you are applying the patch to a unix system you're going to have to run the dos2unix command on the CreateOIDContainer.sh file in the <OH>/discoverer/util directory. Also, notice the CreateOIDContainer.sh file, yes its case sensitive and don't forget to edit the file and replace the %ORACLE_HOME% directive with your actual Oracle Home path, while you're at it make sure you chmod 750 CreateOIDContainer.sh too. Can you tell it was a windows guy who wrote the script
.
Anyways, before executing the CreateOIDContainer.sh script you'll also need to properly setup all your paths which includes:
export ORACLE_HOME=<Your Oracle Home Path>
export LIBPATH=$ORACLE_HOME/lib32:$ORACLE_HOME/lib:$ORACLE_HOME/jlib:$LIBPATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib32:$ORACLE_HOME/lib:$ORACLE_HOME/jlib:$LD_LIBRARY_PATH
The big kicker that bit me this last time was that I was getting an error that the Discoverer product container doesn't exist in the OID Repository. Well, for some reason that container doesn't actually exist until a user is created in the OID through something like OIDDAS, since this client was using Server Chaining to bring in all their users (another blog post soon) we had never actually created a "new" native user in OID.
Needless to say the 10.1.2.3 Discoverer patch set can't come soon enough which includes this patch with it. Till then feel free to drop me a line if you have any questions. If you want some more in depth examples of the config let me know and I'll get some screen shots the next time I do this.
Ora-Click-Clack Weekly Review #2
Alright, last night I posted on Twitter that this would be released this morning, well its still morning somewhere. Without further ado here is version two of the click-clack. Make sure you read through to the end for some interesting changes coming to the site soon or already deployed.
- The first story was submitted by me, and not through the big controversial RSS feed option. The story entitled Do you recognize the man in this picture? came to my through my normal RSS feeds. Mashable is a great site that covers new and exciting web 2.0 site and other technology news. Low and behold I'm going through their feed today and saw an article that sparked my interest on a judge who sentenced a hacker for listing the ips coming from a usenet server. I clicked through and look at the picture in the article and its none other than infamous Oracle DBA / Miniature Horseman / Truffle Entrepreneur Don Burleson. Yes that is truly Don, if you don't believe me, scroll down to the bottom of his own website for the picture. Too funny. 1,000,000 bonus points for Dan Norris for being the first to identify him.
- The next article comes from Patrick Wolf the Oracle ApEx world asking people if its a good thing to create an ApEx certification. Certifications are something that I've always debated about with people, many people can read a book, pass a certification test and have no idea how to apply things in the real world. While there are some certifications that combine book and hands-on during the tests many do not. Personally, while certifiable, I don't carry any certs its just something that doesn't appeal to me, I prefer to spend my time learning new things and sharing with the world letting me experience and reference speak for themselves. I do agree that for newbies, certifications provide a great way for them
Ora-Click-Clack Weekly Review #1
Well, part of starting Ora-Click I wanted to start a series of blogs covering the top 5 articles of the week. Due to some technical difficulties I'm a week late, but I promise to make it extra extra interesting. So without further ado here are the top 5.
- The big story of the week wasn't Jakes post on the "8 Things" but actually the User Your Nose to Install an Oracle Database article. Some people claim that Oracle software is among the hardest to install. I tend to agree many of the earlier releases (<8) were a pain in the butt, but Oracle's come a long way since then. Just to prove it Howard posted a link showing exactly how easy it is. I would love to link to it, but as I'll discuss later Howard has shut down his blog until the "8 Things" craze slows down. When he gets back to earth the article can be found here.
- The second article covers the new fixes and features of the latest mix.oracle.com release. I'm proud to be member 41 on the site and really think its a great way to get open and honest customer feedback. As the site grows there's definitely some new features that need to be added to help manage consuming all the new content but knowing the AppsLab guys they're already working on it. I would love to see Oracle open up the source to mix, as suggested by Jake here. I know that Anthony and Rich are overloaded fixing bugs and working on new features, it would be great to have the Oracle
Quick Post : Oracle jDeveloper Technical Preview 3 Virtual Machine
Just a quick post on something I was working on, my father passed away suddenly yesterday so I don't have as much time to put into this post as I had hoped. Anyways, over the holiday I put together a quick Ubuntu VMWare virtual machine with Oracle XE, the new 11g Technical Preview 3 jDeveloper release and the SOA suite. Over the last week I uploaded it to Amazon S3 and am now distributing it as a torrent. Here are the links to the four torrent files (Amazon doesn't let you compress multiple files into a single torrent).
http://www.matttopper.com/personal/torrents/Oracle11gTechnicalPreview.disk1.zip.torrent
http://www.matttopper.com/personal/torrents/Oracle11gTechnicalPreview.disk2.zip.torrent
http://www.matttopper.com/personal/torrents/Oracle11gTechnicalPreview.disk3.zip.torrent
http://www.matttopper.com/personal/torrents/Oracle11gTechnicalPreview.disk4.zip.torrent
It is a VMWare virtual machine, personally I still like VMWare because I can run it on my laptop and take it with me anywhere. Oracle VM has its place in the server room, but not on the desktop, yet. It can be used by anyone with VMWare Player (www.vmware.com) for free.
The username and password for the machine is : oracle / oraclevms
These are torrents so please continue to seed them after your downloads are complete, this is the only way I'll be able to afford to keep them online for people. If you are new to torrents check out http://support.bittorrent.com/ there you will find all the info you need, or just ask one of your pirate friends and I'm sure they can help you out. Admittedly the VM isn't as nicely put together or as small as I normally do, but due to the circumstances I hope people can understand. I may be offline for a few days, but enjoy the download. This is my trial run for the bigger Oracle/VMware torrent site I hope to have together soon. Thanks to everyone helping with it, we'll be launched shortly.
Oh yeah, and don't forget to "Click It" below: ![]()
Announcing Ora-Click.com – The Digg Site for Oracle Stuff
I like to think of myself as a pretty good follower of the Oracle news world, I spend at least an hour a day going through blogs, tweets, google alerts, etc. to make sure I've got a finger on the pulse of everything going on at Oracle. Honestly, its never enough, and for the many Oracle customers who's job isn't to keep up with every piece of Oracle technology its a confusing world right now.
Every other week we are seeing a new acquisition, new great code examples, etc. and there is no time to keep up with it. Also, its hard to find the best content through all the different sources. I've been using sites like Digg and Reddit to keep up with non-Oracle news. While Oracle is huge there is no way Digg would ever create an Oracle category, so I began my search to create my own Digg clone for the Oracle community. Luckily someone already had the same type of idea and created the open source software package, Pligg. All I had to do now was come up with a new design, deploy the site and tweak some of the settings.
So without further ado here it is http://www.ora-click.com the Digg clone for Oracle news. Feel free to join the site, submit some stories and contribute some comments. Right now I've got it set so that it takes three "Clicks" for it to appear as Popular News, obviously over time as more stories are submitted we'll have to bump it up, but I think three is a great start.
If you're looking to submit your own news all you have to do is log in and go to http://www.ora-click.com/submit.php put in your URL, story title and description. If you're looking to embed the Ora-Click buttons in your own site I've created two sets of buttons to add to your posts.
The small button:
which can be added with the code:
<script src="http://www.ora-click.com/clickit/button-small.php" type="text/javascript"></script>
And the large button:
which can be added with the code
<script src="http://www.ora-click.com/clickit/button.php" type="text/javascript"></script>
It's that simple, feel free to Click this story and try it out. The site has full RSS feeds for the entire site and also the categories individually. If you have any questions or comments please let me know either through comments here or via email at matt@matttopper.com.
Have fun and enjoy, for those that have asked the virtual machines site isn't dead, I've just been working a million hours and haven't had time to bring it up to my standards. Look for me to launch a new VM tomorrow on this site as a trial run for it.
Twitter Tracking
A little while ago (ok about 2 weeks) Twitter released a new feature that lets you track "concepts." The idea is that you can track anytime anyone, not just the people you follow, mentions a word or series of words. That way if you want to see anytime anyone posts something with the word "Oracle" in it or "Matt Topper" you will immediately get an update. While that's cool, what more interests me is the model they are implementing it with. I'd really like to see the event model and code they are using to filter every post that is made. Holy overhead batman!! Anyways, if you want to start tracking stuff, all you need to do is text your Twitter number with the message "Track <topic here>" i.e. "Track Oracle". You can see their blog post here: http://twitter.com/blog/2007/09/tracking-twitter.html and if you want to track me I'm over there at http://twitter.com/topperge
OK, back to my install fest weekend. (and playing with RFID on the side)
Oracle Adaptive Access Manager : Day X : Suspended…
Well, I spent a good amount of time last night installing and configuring the OAAM software, getting it up and running, configuring the data sources, etc. and came to the point where you have log in. Hmm...well...the docs don't have a default user name and password to use after the install. So being Oracle we all know where the default user name is at least stored, in the database somewhere right? Wrong, all the user tables were empty for the product. So at this point I'm stuck, I've got an SR open, but being such a new product I doubt that will get me far. I've also got a couple emails into people I know internally at Oracle so hopefully Monday I'll get some updates. In the meantime I'm going to work on some slides for my upcoming Oracle World presentations and finish up some virtual machines and get them uploaded to the OracleVMs.com site. S3 has turned out to be a pain to get everything working, but I'm close, hopefully I'll have a release by the end of the weekend.
Oracle Adaptive Access Manager : Day 1 : The Database Config and Install
Lets start with a little overview of the OAAM Architecture. The software is made up of two components:
- OASA - Oracle Adaptive Strong Authenticator
- OARM - Oracle Adaptive Risk Manager
The OASA component is provides numerous means to protect from phishing, pharming, trojan, and keylogging attacks. The different authentication options are interesting and well designed to protect an organization. The OARM component tracks user usage patterns to help prevent fraud detection. It uses such things as your user name, known computer attributes, known locations you access the system from, etc. to build a comprehensive profile about you. It also allows organizations to monitor in real time what users are accessing their systems and either automatically or manually (through an operator) force a user to perform a secondary level of authentication to prove they are the actual person at the computer. The Risk Manager creates a virtual fingerprint of your users through a combination of IP addresses, geo location, city, state, country information, connection type, connection speed, routes, isp flag, ASN, carrier, top level domain, registering organization, hostnames, routers, and how quickly you navigate pages. You never realized there was so much stuff people could track you by and you don't really have an option of controlling it.
OAAM is deployed just like any standard J2EE application. It's packaged as a set of jar files, some static images, and database scripts with property files to link it all together. Nothing here that I haven't done a million times before. So lets start at the bottom and work our way up. I started with a standard Oracle 10.1.4 OID /SSO deployment and 10.1.3 OC4J/OHS install. So onto the database.
The database much be of version 9.2.0.4 or higher, honestly, if you're deploying a new software architecture and still using 9.2.0.4 database to store your data give me a call and I'd love to discuss the business reasons for sticking with that. Lucky for me, the basic OID install uses a 10.1.0.5 database so I'm ready.
The first step is to validate the database parameters so that they match the minimum requirements for the install. We first must get access to the pile so we can change it into an spfile to open the database. So first we log into the database as sys.
After we log into the database, create a pfile from the database parameters, after the pfile is created you need to then edit the init<sid>.ora file you've created. To do this we set our ORACLE_HOME, ORACLE_SID, and PATH.
$> export ORACLE_HOME=/software/oracle/oaam_oid_1014
$> export ORACLE_SID=oaamoid
$> export PATH=$ORACLE_HOME/bin:$PATH
$> sqlplus /nolog
sql> connect /as sysdba
sql> create pfile = 'initoaamoid.ora' from spfile;
sql> quit;
$> vi $ORACLE_HOME/dbs/initoaamoid.ora
Within the pfile you need to set the minimums for the following settings:
*.db_writer_processes=4
*.fast_start_mttr_target=300
*._hash_join_enabled=TRUE
*.open_cursors=2000
*.pga_aggregate_target=512M
*.processes=500
*.query_rewrite_enabled='FALSE'
*.sessions=500
*.shared_pool_size=500M
*.star_transformation_enabled='FALSE'
*.timed_statistics=TRUE
*.undo_management='AUTO'
*.undo_retention='3600'
*.session_cached_cursors=500
*.commit_write='BATCH,NOWAIT'
One thing to note, commit_write is not a 10g R1 parameter, it was introduced in 10gR2, so the instructions are a bit off, but not hard to get around, just don't put it in the end of your pfile. Write out your pfile to the OS and finally we log back into the database, create the spfile from the pfile and start the database.
$> sqlplus /nolog
sql> connect /as sysdba
sql> create spfile = 'spfileoaamoid.ora' from pfile;
sql> startup
sql> quit
Once the database is started we then have to load all of the data that allows the components to run. To do this we log back into the database and then execute the db_setup.sql script.
$> sqlplus system/<oraclevms>
sql> @/software/source/oracle/AdaptiveAccessManager10g/oracle_fa_database/db_setup.sql;
The script asks you to enter the location where you want to put the two new tablespaces the application creates, the name of the user you want to use, and the password, then halfway through the script it asks you the name of your temp tablespace (TEMP). Once the script started executing it creates the actual tablespaces, database user, database objects, loads the initial data, and then the location database (for the ip address locations).
Alright, that was stupid simple, the database is done, stay tuned for the software deployment. I apologize for no pictures on this one, I was using a different machine than the one I normally do and my copy of SnagIt was setup to only snap one screen shot and overwrite it each time. I promise the rest of the install log will have pictures to go along. Thats all for now, check back in a few hours (maybe 24) for the java application deployment instructions.




