Computer Vision

[Sama]

This is what this project aims to do: (not all threads may make full sense as they were pasted in. if they don't, ask)



Here's a comptuer vision primer.

And here's a sticky link to the latest code: CarVision_0.4a.jar. This also contains the source code.

If you don't know anything about coding but would like to see how things are developing, then here are the instructions on how to install the progam posted below.

First, you need to get a Java Runtime Environment installed. Go to http://java.sun.com and download the latest for your operating system.

Then, you'll need the Java Media Framework. Get that for your OS from here

ensure that your webcam/video capture card is working first, without Java. So use whatever program you normally use to do bi-directional porn and make sure the video feed is working.

Now that you have all the software, start JMStudio, which JMF should have just installed, and go to File -> Preferences. Click on the Capture Devices tab, and click on Detect Capture Devices. This will register any capture devices you have so that Java can see them. Just make sure the VFW (video for windows) one is visible, and that it's locator is vfw://0 (I'll add the ability to select which locator to use later).

Finally, double click on the jar and you should get a window appear, which will immediately start to capture the background. Do not stand in it's way whilst it's learning. Let it point at some part of the room/road for a few seconds so it can distinguish between background and foreground.

The result will be that any moving objects should only appear, whilst non-moving objects will remain black.

This currently works best indoors, or on a day with consistant lighting.


Orignal post

I'm developing a computer vision library to later create applications with. The Java Media Framework and Advanced Imagine Api's are being used to extract frames from a stream and process the pixel data.

So far, I have exactly that. The next steps are to create algorithms for motion detection and the likes. If you have any experience in the field and would like to get involved, please get in touch.

[Sama]

So far, I have the following algorithms that I've done in different projects. I'm currently in the processes of grouping them into a library.

1. A motion detector that continuously learns a background, and subtracts the current frame from that background. This difference image shows moving objects. This image is then scanned using a connected components algorithm, which groups connected pixels together, and labels them as unique regions.

2. I've also got a tracker, that will predict (in a linear fashion) the next location of a moving region from the above region identifier. It uses the centroid of the as the coordinates.

3. I've got code to connect to a video source, either file/url/capture (currently video for windows... not the best but ok for now. directshow would be better). Each frame that is captured fires off an event, which a connected listner can use to retrieve that frame.

the projects I've got in mind are these:

Buffered Recording:
This project is the primary directive. To continuously record, and discard any footage older than a specified timeframe. The idea is that if something happens, like an accident or similar, the user can press a button and have a record of what just happened, in addition to everything that happens after that, up to a specified timeframe / when the user aborts.

Since this stream is running all the time, it can be used for other things. There's a massive distniction between moving and stationary projects, so here's the stationary one first:

Motion Initiated Recording:
This project is failry easy, and can currently be done through various means, but this would put the library to the test. It's to record only when a certan sized region appears in view. The regions can be profiled to distinguish between person/car/animal, using some matching techniques, which will require some research and development. I'm aware that some users will have power issues, but it would be useful to have a recorder run whilst away from the car in a dodgy area for example.

Rain Sensor:
This is sort of both moving and stationary! Using some filters, it should be possible to identify rain drops on the screen. And after a certain threshold, it would be possible to fire off an event to turn the wipres on.

A Passive Radar:
This will be a real challange. A rear view camera that alerts the user of fast moving objects. Since the vehicle is in motion, object detection takes a whole different direction. I'm assuming that optic flow will have to be used here see (this link) for more info.

[Sama]

ok, I've managed to get some code up and running it's attached.

inside the zip, you'll find a jar and the source code.

you'll need to install the java media framework first, and have a device capture card working. you can do this by running JMStudio (when JMF is installed, you get this) and going to File->Preferences. Under the "Capture Devices" tab, press "Detect Capture Devices" (can take a while), then look at the location of that device. it should be something like: vfw://0. commit and exit that screen then try File->Capture. If you can capture in JMStudio, and the location is vfw://0 then the jar application should just work. If the capture doesn't work, sort that out

If the location is different, then you'll need to edit the ProcessorTest.java and change the location.



about the code:

currently, it's not optimised or anything, but it does get you familiar and started with grabbing and processing.

If you leave the camera pointed in a direction, it will slowly learn that background. After about 10 seconds, try to move around in front of the camera. Only the moving parts should appear in the image (plus a load of noise, depending on the quality of the cam, but you always get noise).

Be careful not to make the camera's auto-intensity change too much, by going too close to it, since this will fool the algorithm into thinking the background has changed.

have a mess around with this and any questions, fire away.

[Sama]

the code should makes some sense if read in conjunction with computer vision primer.

it sounds like you're looking at the image learnt rather than the foreground. I'm not sure if you messed with what is displayed. when you move in front of the camera, do you appear at all? check these in ProcessorTest:


// image = motionDetector.getBackground();
image = motionDetector.getForeground();

if you view the background, that'll show you the learnt model, so you can see it's learning correctly. the foreground will be the current frame minus the background (+ noise and other light variations)

[Sama]

this new version should work much faster and much more accurate. just let it learn the background for a good 30 seconds before you add anything.

Due to the background model being a running average, the frames learnt to begin with take a lot longer to "forget". So if there are objects that are in the frame to begin with and move after a few seconds, they linger in the model for a while before they're unlearnt.

I'd like to add a weighting method of some sort that gives newer frames higher value in the sample set.

I've discovered that for b&w, we only need intensity, so hue and saturation of colours can be ignored.

moreoever, the output can be colour (if the input was colour), even though the algorithm is working in b&w mode. the reason for this is that the pixel values of the foreground are copied in their full colour from the orignal image, but they are detected as intensity changes only from a b&w image. pretty cool eh?

anyhoos, have a mess around with this latest version when you get the chance. it's refactored and responsiblities of classes are much better seperated now, so it should be easier to grasp. I've still got more to do when I get the chance to.

some words of warning:
1) if the environmental lighting changes, the background model will be useless and will have to be learnt again.
2) the RGB mode sucks at the moment due to some bug. in general, HSI is a better model to use so looks like I'll focus on that for now, until RGB is required.

have a look at the constructor of ProcessorTest for some tweaking.

here's an image of the algorithm seperating my hand+water bottle from the rest of the room:



it's a bit choppy, but that's the purpose of the next part of the algorithm

coming soon:

1) smoothing prior to analysis - reduces the number of holes in the foreground
2) region analysis - grouping pixels that touch and placing a rectangle around them (with an id number)
3) region tracking - to make sure the id number stays associated with it's region
4) multimodal backgrounds - multiple background models based on different conditions. nighttime/day/overcast/sunny with autoswitching.

these will be very CPU intensive... but we'll see how it all pans out

all the above is still for stationary use. analysis whilst moving is completely new to me and I really need help there!

[Sama]

here's some small changes so that now it can clearly be seen how the algorithm is working:



for each pixel in the stream, a running mean is calculated. this is only for intensity and therefore is b&w as can be seen in the background image.

for each pixel in the stream, the difference between the background and the current value is caculated. That is:

SQRT[[meanPixelValue - currentPixelValue] * [meanPixelValue - currentPixelValue]] - a square root of a squared value ensures it's always positive

so now that we have the mean and variance, we say this (for each pixel):

            Mean
|------------|------------|
 < (-v * d) > < (+v * d) > 

where:
  v = variance
  d = deviation

if the new pixel value is within the mean and its variance, then it's a background pixel so ignore. if it's outside the mean and its variance, then it's a foreground pixel so extract it.

the variance is multiplied by the deviation, so the more deviation, the more variance is allowed.

not difficult at all!

[Sama]

for linux:

try this jmf download for linux. I've never tried it on there, but that's the only part that needs to be working. once you have a capture device working with the JMF, this code should work failry easily.

[MiT]

Wow... you sure are doing alot of stuff. Keep it up. maybe one day i can make some use of it.

[Sama]

that's what it's here for!