populate database from external XML or Json or Webservice?

Problems with WiFi, SQLite ,Bluetooth, WiMax, Proxies, etc...

populate database from external XML or Json or Webservice?

Postby achie1266 » Tue Nov 10, 2009 8:21 pm

Hi,

I just started Android development and am building an app which should use data from a server and store some of it in the database.

Lets say I have around 4000 people and 1000 employees and the total data that after putting into sqlite database comes to around 10MB.

What would be the best approach to follow?
1) Download the database itself and store it on the phone and use it? If so how would I save the database on the phone?
2) Read data from an XML file on the external server?
3) Read data from a Json file on the server?
4) Are webservices better for this?
5) Is there any other better way to do this?

Can anyone please weigh these conditions. Which one will perform better and faster?

thank you.
achie
User avatar
achie1266
Master Developer
Master Developer
 
Posts: 223
Joined: Mon Nov 09, 2009 10:56 pm
Location: Denver

Top

Postby divestoclimb » Thu Nov 19, 2009 6:52 pm

10 MB is a lot of data to download over a phone connection all at once. Your best bet is to have the app consume the data on-demand from web services and cache its results locally in a database. If the entire database must be accessible without a data connection, then of course that's not an option.

Your options for transferring that data to the client should be restricted to formats that don't require a large amount of overhead in the protocol. That rules out XML ("<tag>data</tag>" has only 26% actual data before compression), and even JSON isn't much better. Maybe Hessian Binary Web Service Protocol?

If you must have the entire database on the phone, I recommend you build a backend that lets you download the whole dataset once, then also build web services that let you resynchronize the set so you don't have to re-download the whole thing any time one or two records change.

For storage on the phone, you probably want to build and maintain a SQLite database on the SD card; 10 MB is too much data to practically fit in the internal memory of a G1, and even on newer phones it's a significant percentage of the total storage space. Using the SD card will introduce the potential for complications if the card is unavailable or not safely removed. The ideal solution, but slightly more complicated, is to let the user pick where to store the database: on internal memory or on the SD card.
divestoclimb - Android SCUBA app developer
Android Community | Gas Mixer, ScubaLib, Scuba Tanks | d2c Checklist
All code is open source under Apache license.
divestoclimb
Developer
Developer
 
Posts: 33
Joined: Mon May 11, 2009 7:46 pm

Postby achie1266 » Fri Nov 20, 2009 12:31 am

divestoclimb wrote:10 MB is a lot of data to download over a phone connection all at once. Your best bet is to have the app consume the data on-demand from web services and cache its results locally in a database. If the entire database must be accessible without a data connection, then of course that's not an option.

Your options for transferring that data to the client should be restricted to formats that don't require a large amount of overhead in the protocol. That rules out XML ("<tag>data</tag>" has only 26% actual data before compression), and even JSON isn't much better. Maybe Hessian Binary Web Service Protocol?

If you must have the entire database on the phone, I recommend you build a backend that lets you download the whole dataset once, then also build web services that let you resynchronize the set so you don't have to re-download the whole thing any time one or two records change.

For storage on the phone, you probably want to build and maintain a SQLite database on the SD card; 10 MB is too much data to practically fit in the internal memory of a G1, and even on newer phones it's a significant percentage of the total storage space. Using the SD card will introduce the potential for complications if the card is unavailable or not safely removed. The ideal solution, but slightly more complicated, is to let the user pick where to store the database: on internal memory or on the SD card.


Thank you, I have been waiting for someone to reply me about this one for a while.
Any way I decided to not package entire data with the app but I am using xml to download the data when user opens the app and update the data whenever required.
thankd for mentioning about the Hessian Binary Web Service Protocol too. I don't have any idea what it is but is sounds to be very good light weight protocol. I will look into it later if I need web services.

Now that I am trying to download data from external xml, can I also instead package my sqlite database files with the application and use them in a secure way? I would like to store it in a default place where the application searches automatically for database.

Thank you.
achie
User avatar
achie1266
Master Developer
Master Developer
 
Posts: 223
Joined: Mon Nov 09, 2009 10:56 pm
Location: Denver

Top

Return to Networking & Database Problems

Who is online

Users browsing this forum: No registered users and 5 guests