Cliff Niersbach NAR response to Google spidering listings in Mibor
Posted May 6, 2009 @ 2:51 pm, Viewed by 2942 Visitors, Read 3760 Times.Many of you read over at Agent Genius about how MIBOR (The local board of Realtors in Indianapolis) has demanded that all IDX vendors stop Google and other search engines from spidering any page of a Realtors website that has IDX listings or any part of the data referencing listings (such as property descriptions, MLS #'s or addresses) - this is with the exception of just the agents own listings of course.
Naturally as one of the largest custom IDX vendor in the world we are more than concerned about this issue as should all Realtors as it has the potential to dramatically impact their ability to get their listings competitive exposure unless they go through a third party aggregator such as Realtor.com and pay through the nose for premium advertising.
Also as the IDX vendor for Paula Henry http://www.hometoindy.com (the agent who first blogged about this) and the IDX vendor for her broker Mike Taylor http://www.reddoorindy.com I have first hand knowledge of the situation, in fact I am the person at REW that was handling the communication back and forth between NAR, REW, our client and MIBOR.
In the end - both NAR and MIBOR failed to see that the complaint lodged and the decision they were making had incredibly serious consequences that only served to do harm to their members and the listing clients they represent.
Over the next few days I am going to be documenting the communication between myself, NAR, MIBOR and my client (I have already spoken to him over the phone and he supports me in this decision) in hopes that we can raise enough awareness regarding this issue and have it nipped in the bud before this local (board level) situation spreads to other ill informed / uneducated boards.
I wrote:
"Mr Nierbach (if you would be so kind) can you please address "Google" specifically and your policy regarding MLS listings
Mr Renkert has asserted that it is your position that Google may not spider any MLS listed property - is this in fact correct?
He also asserts that it is ok for Google to have MLS listings spidered as long as the broker has placed the listings on the site being spidered (example Trulia) which is a contradiction. It sounds like he does NOT have a problem with Google indexing listings rather he wants to be able to dictate how they get there.
The bigger issue (I beleive) is whether or not this policy (as it pertains to the single largest source of internet business for brokers - ie Google) has been interpretted for this particular situation. It most certainly is not being universally communicated or enforced as evidenced by the hundreds of thousands if not millions of agent websites, broker websites, IDX vendor websites and third party listing sites that currently allow (or rather encourage) Google to visit all of their inventory pages in an attempt to drive their traffic to the listing they have searched for.
So again - if brokers ONLY wanted their listings to be able to be accessed at thier website (this is the road we are going down) why have IDX in the first place?
I do hope this issue can be discussed (and resolved) quietly as I really think that it has the potential to become something a lot bigger than it needs to be. The solution - (rather a solution) do not consider major search engines (Google, Yahoo, Live etc) "scraper sites" (because they truly are not) and thus the rule being quoted would not (and should not) apply. "
To which mr Niersbach replied
"Good afternoon Mr. Carey:
I respectfully advise NAR policy does not expressly focus on Google or any search engine operator in particular but rather on the rights and obligations of MLS Participants when they are afforded the privilege of advertising other Participants' listings pursuant to the IDX policy.
Real estate brokers are prohibited under the Code of Ethics and under the licensing laws or regulations of most states from advertising other brokers' listings . Under IDX, listing brokers can, but are not required to, give other participants authority to advertise the listing broker's listings on the other participants' websites. Absent the listing broker's consent, other participants have no authority to advertise those listings in any medium, including the Internet.
When an MLS Participant who is not the listing broker, pursuant to a grant of authority from the listing broker, advertises one or more of the listing broker's listings on their website, he or she does so subject to the IDX-specific rules of the MLS which they participate in. Those rules are based on the IDX policy of the National Association of REALTORS®.
The IDX policy provides, in relevant part, that "Participants must protect IDX information from misappropriation by employing reasonable efforts to monitor and prevent 'scraping' or other unauthorized accessing, reproduction, or other use of the MLS database." This is the provision referenced in my earlier correspondence with the Metropolitan Indianapolis Board. Simply put, when an MLS participant advertises another participant's listing pursuant to IDX, the participant doing the advertising must take reasonable steps to prevent that other participant's listing from being scraped. It is important to note that this requirement applies only to the listings of other participants. The website operator is free to do whatever he or she chooses to do with his or her own listings, including allowing their own listings to be scraped, published on third party aggregator sites, etc.
Cliff Niersbach "
To which I responded
"Cliff,
Thanks for your response - I am in fact aware of the IDX policy and specifically to the prohibition of advertising another brokers listings anywhere but on the IDX approved brokers website. I am also aware of the requirement of us (the IDX vendor) to employ appropriate data protection measures to ensure that data is not "scraped" - by "scraper sites"
The "NEW" arguement here seems to be that Google (a search engine) is being lumped in as a scraper site - and our contention is that Google is NOT a scraper site, rather a technology (just like a browser) with a significant user base with which the public access websites. Google simply tells the user where to go - they are not creating web pages of their own out of the data they index. I am sorry, but I just simply do not see how NAR could consider a search engine to fit under the same designation as a "scraper site" -
Comments? "
And then Cliff from NAR replied with
"Good afternoon Mr. Carey:
As noted in my earlier email, the NAR policy requires that participants displaying other particpants' listings on their websites make reasonable efforts to prevent scraping. It does not limit the obligation to prevent scraping to scraping done by entities you refer to as "scraper sites", but rather applies to scraping in general.
I believe the issue isn't whether MIBOR is characterizing Google or any other search engine as a "scraper site". Rather, the issue appears to be whether any information provided by other participants and displayed on an IDX site can be scraped by anyone.
I hope this responds to your request.
Cliff Niersbach
And then finally - this exchange:
Good evening Mr. Carey:
Please find my responses to your questions and comments below. I appreciate your follow-up and the opportunity to respond.
By copy of this email, I will share your concerns with the REALTOR® leadership of the NAR Multiple Listing Issues and Policies Committee. I will also review your comments and my responses with the head of the Center for REALTOR® Technology ("CRT") and with the NAR General Counsel. If they have different opinions, I will follow up with you and with the Metropolitan Indianapolis Board.
Cliff,
Perhaps I am not phrasing my question correctly - I will pear it down to as simple a form as I can:
Does NAR consider "indexing by search engines" to be the same as "scraping"
Response: The CRT advises that "indexing" is a form of scraping.
Because if it doesn't then the rule being cited does not apply.
We at Real Estate Webmasters do not feel that it is the same thing nor does our customer Mike Taylor (NAR and MIBOR member)
A "scraper site" stores and publishes content on their domain generally for the purpose of misleading or profitting from the publication of specific data that has been manipulated (generally) in some way.
A "search engine" indexes the location of data and "directs users to the original author of the content" which is not advertising, or malicious in any way. Nor is there any manipulation of data in the indexing process, it's simply a way for that particular service (in this case Google) to answer the question of "where is x" for it's users.
Response: NAR's IDX policy doesn't distinguish between "malicious" scraping and scraping that might (for lack of a better word) be described as non-malicious or even beneficial.
This is an incredibly important issue to address as we feel that the particular rule being cited out of context and is being used innappropriately as a "catch all" and interpreted and enforced arbitrarily to discriminate against our client. This board is NOT enforcing their "rule" against all members equally nor is NAR enforcing this rule (with respect to search engine indexing) anywhere else on the internet (there are literally hundreds of thousands of NAR member websites with IDX's that are indexable by search engines) - are you saying that all of these websites must be changed?
Response: The Metropolitan Indianapolis Board has a long and deserved reputation for integrity and fairness. If a REALTOR® has concerns about MIBOR's rules, or how those rules are enforced, the appropriate step is to engage the staff and REALTOR® leadership in constructive dialogue.
The fact of the matter is - NAR has an obligation to enforce it's rules fairly and accurately.
Response: Enforcement of MLS rules is the responsibility of local associations of REALTORS® and their MLSs, NAR's role is developing policies and procedures that permit MLSs to serve their participants in effective, efficient and legally defensible ways.
We understand that and take no issue with the "protect against scraping" clause - in fact we agree with it and actively participate in the defense against scraper sites (we don't like being scraped either)
NAR has an obligation to it's members to protect their rights and freedoms and ensure that they are treated fairly and equally and defend against discrimination (one issue at stake here)
Response: NAR understands that responsibility. NAR's concern for its members is manifested in the body of MLS policy that has been developed and implemented over the past four decades.
NAR finally and perhaps most importantly has an obligation to protect the interest of the seller. To deny a seller representation of their properties in search engines is to deny them one of the most significant opportunities to sell their home via a qualified professional. If NAR were to make the huge mistake of stating that search engines should be blocked from indexing IDX participant websites then the only results left (because it won't stop users from searching for them) will be the very scraper sites that you are trying to defend the public against be exposed to.
Response: I respectfully disagree with this conclusion. The basis of seller representation is the exclusive listing agreement that sellers enter into with listing brokers. It is those listing agreements that spell out whether the listing will, for example, be submitted to MLS; whether information about the seller's property will be submitted to third-party aggregator sites like Realtor.com; whether other brokers will be afforded the courtesy of displaying information about the seller's property on their sites, etc, As a practical matter, the fact that a search engine cannot index or scrape information a listing broker allows to appear on other brokers' sites to then direct consumers to those other brokers' IDX sites doesn't alter the fact that consumers viewing those sites will have access to every listing on those sites. The purpose of the IDX policy is to give listing brokers a method to consent to display of their listings by other participants on their websites. Absent such consent, advertising (whether on websites or otherwise) by other brokers is prohibited by the Code of Ethics and by the license laws of many states.
But websites that scrape content for their own gain or malicious purposes are not the same as search engines and should not be mentioned together in the same sentence - they are different and this needs to be acknowledged by NAR. I humbly request that you consider this issue and offer a final decision in this matter allowing participants to have their websites spidered by search engines if they so choose.
Response: I can only reiterate that NAR's IDX policy mentions neither websites that scrape for their own gain or for malicious purposes - or search engines that scrape for other reasons. I agree "scraper sites" and search engines are different - and must again point out that neither has any authority to scrape listing brokers' information from other brokers' IDX sites.
That said, as indicated above I will review your concerns with the head of the CRT and the General Counsel and if I have misstated the IDX policy in any way, I will respond to you and to the MIBOR. I hope that you and the leadership of the MIBOR realize that NAR staff has no authority to modify policies established by the NAR Board of Directors. If the IDX policy - or, for that matter, any other NAR policy - requires clarification or amendment, that would require action by the Board of Directors.
Please let me know if you have other questions.
If you have a question or comment regarding my blog entry, please use the comments box below - if you have a general question about REW Blogs or would like to discuss any facet of being successful online, please visit the forums.
Rate this Post!
Share this Post
Print
Social Bookmarking78 Responses to “Cliff Niersbach NAR response to Google spidering listings in Mibor”
Zillow, Trulia and Realtor.com will not be affected by this - Zillow and Trulia because they do not recieve IDX feeds and Realtor.com because they do not have to play by the local boards rules.
I'm on my regional MLS board - haven't been there long but one of the tests of "reasonableness" that they use is if the national sites that don't have to follow any specific rules allow something - does that get them an unfair advantage - if it does than they have been willing to change some of their rules to allow local agents to compete with these national sites. I doubt this is how every MLS board operates but that is what I have noticed with mine.
This issue is very disturbing to me.
Hi Carolyn,
That is my point, - Zillow, etc., will not be affected, therefore they will be the only websites indexing IDX listings as agents give them permission to do so. These power houses will be capturing the longtail searches while the rest of us are only able to allow Google to index our personal listings.
Joe 
Paula Henry
Carolyn - I would love to see the test of reasonableness at my local board, indeed, quite the opposite. They threatened to zap our feed before the stamp was on the envelope.
If we do not control the data we provide, we will be at the mercy of third party aggregators, Realtor.com and the true scrapers. To think our board and NAR believe Google is a scrapper speaks volumes about the ineffectiveness of our industry leaders to stay up to date on technology.
Unfortunately I wrote a long and well thought out response to this issue and to MIBOR, only to have it disappear when I entered the security code as I had been timed out. Ouch. Writing is never as good the 2nd time.
I posted a blog entry on ActiveRain about this and it's now featured on AR's homepage. I'll include a link in that blog entry to this REW blog entry. Hopefully this thing will gain some momentum. 
Jugding from the lack of comments, I don't think people get how serious this situation really is! This has HUGE implications for all of us who have/had search engine friendly IDX's.
Thanks to REW and their awesome custom IDX, Paula's and my website are outranking our competitors for many long tail search items. Rather than actually learn something about internet marketing and actually service his listings, our competitor decided to just file a complaint with the local board. As you can see, not only did our local board agree, but NAR has backed them on this "Google is a scaper site" idea.
Morgan, thanks for posting this and shedding light on what is an extremely ridiculous policy that is cutting the legs out from underneath the NAR members. As Joe said, we as NAR members are subject to these rules while the national lead aggregators don't have to abide by any of their arcane rules. Enforcement of this policy only puts agents back to 1970's while giving r.com and the nationional lead aggregrators an unbelievable advantage over the NAR membership at large.
Joe, any reason you made the AR post member's only? It won't get indexed if you do that. Think this issue should be public - just figured it out cause I linked to it from Twitter.
I'll write something up on my blog as well and link to all of the above.
Mike I definitely think this is HUGE - way bigger implications than Sarasota MLS. I totally feel for you - but it's way bigger than your site. It may take a little bit to get some traction-but let's see what we can do to get some help with getting the word out.
This policy is an outrage and even nefarious. I cannot entertain that adults with any heart for consumers would try to state with a straight face that indexing by search engines is an unethical scraping of data. How can MIBOR intentionally try to prevent search engines from indexing the IDX data? How is that serving the marketplace and consumers? Intentionally making it harder for consumers to find the homes for sale for which they are searching is anti-consumer. It is this 20th century mentality of trying to control the market and data that has damaged the reputation of realtors and brokerages in the past.
Are you blind, MIBOR? Are you anti-consumer? Have you actually thought this out?
And what about the sellers? You better believe they want their listing indexed by google and easily found from anywhere. How is MIBOR's position allowing brokers to better serve their sellers? It is consciously making the broker less effective either in exposing the listings or in connecting with and forming relationships with buyers/consumers.
What is also aggravating is that they are intentionally making it harder for agents/brokers who are modernizing their business and business models to be harmonious with the behavior and preferences of today's consumers to connect with and serve more clients. MIBOR is intentionally penalizing those who are on the cutting edge of reaching out to serve consumers most effeciently. What is gained by this control MIBOR is trying to maintain other than a sense of controlling others' businesses and thwarting buyers from finding what they are looking for quickly? How is this serving the marketplace MIBOR? It most decidedly is NOT.
Mike said it well:
"The problem is this ruling gives/ will give an unbelievable boost to lead aggregators like Trulia, Zillow, and our good friend Realtor.com. R.com aside, the lead aggregators are not members of NAR or the local boards and are not subject to the rules they are imposing on its members. So while NAR chooses to cuts its members legs out from underneath them with their archaic IDX blue laws, the national lead aggregators are the real beneficiaries here."
That any MLS board would endeavor to either be this backwards, controlling of their members or anti consumer is alarming and not what our country and capitalism is about. Handcuffing the most creative and entreprenurial is disgusting and depressing and is not what being a realtor or small businessman is about. Why would NAR stand for this abuse? I pay so much money so they can do what for me--handcuff my business through which I feed my 5 children?
MIBOR supposedly is open to reviewing their policy with more input from its members and other interested parties. I hope they are sincere in saying that....Everybody is watching you MIBOR.
And perhaps the obvious solution is that all listings should default into an IDX open to indexing by search engines when they are put into the MLS. I CANNOT IMAGINE 1 SELLER NOT WANTING THEIR LISTING TO BE INDEXED BY GOOGLE. Do you care about that MIBOR? Let's be honest about this anti-consumer policy. If a seller does not want their home exposed on the IDX feed that can be indexed by search engines, they can opt-out when they sign the exclusive right to list contract and the listing agent can opt-out when they list the home in MLS.
Sincerely,
jay
Thanks for pointing that out Carolyn. I just made it public.
Hey Jay, this would be a good comment on your own blog and linking back to the source, even if it were your REW blog. Wanna reconsider where you post this? Might help in the long run. 
What concerns me about this is that our own organizations that we pay money to continue to undermine us. We need these core idx solutions and their full functionality to compete with organizations like trulia for our own business. How is it that other organizations have more rights to agent listings than other agents?
This is a real problem and I suggest that all agents get on board with all of these issues. We are being crippled at every turn and this would be devastating blow to all of us.
We have to fight hard enough against large national sites that are not real estate agent driven; we have to fight for our prospects against companies that do not sell real estate. We need or core IDX solutions and everything we have now to help us stay competitive.
How can trulia, zillow and other companies have more rights to market and how they market real estate listings than me or any other real estate agent? I am not much of an activist, but this should be our real call to arms against all of the injustices we as agents face right now.
This is unbelievable. This is just more proof of how out of touch NAR, state associations and local boards are from the real world. Instead of allowing fair competition, they choose to regulate what is fair and for whom and could care less oabout the end result for the consumer. Seems to me there is a lot of that going around these days everywhere you look. There is a much bigger agenda here. This needs to be exposed and stopped.
MIBOR must be in the stone age when it comes to understanding technology and the Internet and how it relates to real estate! This is truly an outrage. How could it possibly be in the interest of the 84% (or thereabouts) of all buyers who start their search on the internet not to be able to locate listings that interest them? How can it be in the interest of any seller employing a licensed Realtor not to have their listing indexed so it can be found on the Internet? MIBOR must utterly clueless to equate Google with a scraper site! There is something seriously wrong with an organization that cannot find anything better to do than to take up such an ill-conceived cause.
I linked to this post, Agent Genius and Joe's active rain post here:
http://www.itkeepsyourunning.com/mibor-nar-hurting-membership/
I'll keep adding to it as new information comes out.
This situation disgusts me. I can't believe they are considering Google a scraper site. I agree with John about how NAR, local & state associations are out of touch.
I am sure just about every seller out there would love to see their homes promoted on multiple websites versus just their listing agents. A wise listing agent would want the same thing for their client. It seems to me that Mibor is ignoring what is good for the general public and listening to a select few.
Whatever help you guys need let me know.
Should I run for president of NAR? Can I run?
If I were president, you would have my ear. I would be involved in these conversations. We would not have a complacent out touch, out of date organization.
I would also offer programs that encouraged individual agent websites. That folks is what we really need. The agent should not have to compete with other companies for our prospective clients. If every agent had the tools to buy and build their own website, we would in essence be arming the militia, and it is the militia that defeats the strongest of organizations.
Marc - you know how this works - Blog about it (on your blog) share your thoughts and also link to this post, the agent genious post, the AR post , Carolyns post and any other post pertaining to this issue that you feel makes a good case or has good commentary - the more exposure, the more pressure - that's how you can help. Thanks bud (same goes for the rest of you folks)
I know. More or less was asking for a green light.
Btw guys - I would like anyone else who blogs about this submit to post a comment letting me know where their blog is so that I can include it (with a link) to my list of resources / support.
Thanks
The latest is a response from the social media person at NAR Todd Carpenter noting that Paula and Jay Thompson have been invited to testify at the meeting next week in DC on the issue.
"...I have invited both Paula Henry and Jay Thompson to speak before the committee. NAR will be covering their travel expenses as well. Lots of items are on the agenda, this will not be the dominant topic discussed, but Jay and Paula will be able to bring this issue to the committee's attention.
If you are interested in getting more involved, try serving on a committee. Here's a couple links to help you along.
http://budurl.com/serveonacommittee
http://budurl.com/mbrhome"
Boy this is like the Sarasota MLS Board on steroids! What ever became of that travesty?
Just for the record here is what I think of MIBOR and NAR's current actions - http://www.mygulfcoastbeachteam.com/2920292-carmel/
Stop and think about it. Didn't NAR just call Tens of Thousands of Realtors "Scrappers" by siding with MIBOR MLS? NAR is not saying Google is a "Scrapper site" they said they wanted to put programming changes into effect that will prevent Google from reading IDX listings found on an agent's website.
Am I missing something here?
Actually Mike, you are missing something here. I'm not sure where to begin except to say you need to read what's available. My biggest pet peeve is how the decision has the potential of impacting our industry by empowering the big listing houses. If you go to this blog entry,
http://activerain.com/blogsview/1065970/a-local-board-of-realtors-just-gave-more-power-to-zillow-trulia-etc-over-it-s-own-members-
Then scroll down to Lane Bailey's comment. You'll see my sentiments succinctly. Of course, there are other issues as well.
@Joe
I think you have the potential impact of this pretty well nailed. (Nice job IMO) I have posted my thoughts at http://www.ericonsearch.com/mibor-and-nar/ I have not seen Lane Bailey's comment. Will check that in a sec. I thought you said it well...
@Cal, maybe we should have an SEO contest for 2920292-carmel, no?? (grin)
I think Todd Carpenter was out of line about "maybe it wouldn't be so bad..." While I applaud him for getting Paula and Jay to testify next week (I like 'em both), I think that the MLS committee needs to hear from LOTS of folks in the industry as well. The committee needs to REALLY understand the impact of their decisions.
This one is big.
@Eric, that was the idea! Hope you can drive me down in the rankings. Joe Lane too! In fact I hope you guyes push me down to about #200. Looking forward to lots more entries that have a commentary about the situation.
@ Joe, OK. I Understand the point being made and what the ramifications might be with big listing houses getting a huge SEO advantage over local agents and then doling out leads, even if they are our own, at a neat profit to us small frys. Under this scenario the quality of Realty services are going to suffer. So NAR is seriously mistaken if they think this "fix" is quality solution for the public. What it is is a major step backwards on many levels.
MLS belongs to the agents and brokers. We pay to have access to this information and likewise our clients pay us to get the results they expect. Taking away our ability to effectively compete and market online will drive down services and drive up costs.
The point I was trying to make is Google never scrapes, it simply indexes accessable online information and my feelings are as long as BROKER RECIPROCITY exists then IDX data has a right to reside on an agents website (we pay for it!!) and be fully accessable by any search engine. This is actually a benefit to the public to be able to find homes on local agent websites where someone can actually meet with them if needed.
I still maintain that IF the actions being taken by NAR results in Agents being the primary target of their "Anit-Scrapping" rule, then somehow they NAR are talking out of both sides of their mouth. It can't be right for Google to be a "non-scrapper" when it comes to Realtor.com or other "selected" websites which gladly takes business from Google and then be classified as a "scrapper site" when they crawl the website of a well respected local agent. That is a classic double standard that needs to be fixed now.
I believe this is a serious legal issue to say the least. It has the potential to affect the entire real estate trade organization, consumer choice, and the cost of services. Personally it stinks.
knox
Here we go again...
I have placed a call to arms on my blog:
http://arizonarealestatehome.com/real-estate/2920292-carmel
The Great State of Texas defends fair trade and freedom of consumer choice versus market tampering.
Read about it on my blog: http://www.mckinneyhomerebate.com/blog/online-homes-search-technology-heading-back-to-the-ice-age-if-nar-mibor-ruling-stands.html
I have commented on this topic too at http://www.propertyinsantacruz.com/santa-cruz-real-estate-blog/?p=106 as well as at agentgenius.
It does seem our esteemd NAR is once again behind the times technologically speaking. Even being so close to Silicon Valley we still have to print out our contracts even though we can do all the stuff on line. Electronic signatures are so old school, but we don't even have them set up yet, so I guess I should not be surprised that at technology advances there are many that are still back in DOS with their AT machines from IBM While as a Santa Cruz real estate broker and with a firm that embraces technology generally, I am surprised anyone that has issues with the notion that Google is a scraper, is really laughable. Geez, time to hold back on my NAR dues as a political protest!
opps...http://activerain.com/blogsview/1069273/is-google-a-scraper-site-
What next - we won't be able to have our own personal websites? I guess now I don't have to feel so bad that Google isn't indexing (I mean scraping) the listings on my REW IDX. They're indexing some but they dropped thousands. And I understand that my site's authority is the issue with that - I'm just being sarcastic.
Up in arms, everyone - we've gotta war here!
Just for the record - the Framed IDX Solutions get crawled too! http://www.mygulfcoastbeachteam.com/mibor-nar-grasp-internet-technology-works
As Morgan is more business and tech-savvy than most of the MLS board members combined, it's ridiculous that we are even having this discussion in 2009. Whatever happened to free enterprise. In the 1990s, many MLSs thought the Internet would be their undoing, and so did many brokerages. We need to compete and add value, or perish if we don't adapt.
This is exactly why I walked away from the board of Realtors in Atlanta. It's because of the agents who can't adapt to the 21st century of real estate marketing and all their complaining this is going on. What we need nationwide is what we have here in Atlanta like the "FMLS". It is a privately owned company that doesn't have to comply with the NAR or local board of Reatlors. We don't even have to be members of the board.
I have an idea. Why doesn't REW go nationwide and provide MLS services as well? If the sellers authorize the agents in the listings agreements, we can input the listing anywhere we like. The NAR has a choke hold (except in Atlanta) on all the agents who pay the fees. It's disgusting and makes me happy over my choice to NOT be a REALTOR!!!
Paula Henry
Missy's link is not working -
http://activerain.com/blogsview/1069273/is-google-a-scraper-site-
Michael Wurzer
I mentioned this over at AG as well. The error NAR makes in interpreting the rule is that they are reading the term "scrape" independent of the preceding and controlling term "misappropriation". Distinguishing scraping from indexing as a technical matter is difficult (and this is where NAR's technical team went awry in advising NAR staff), but the policy makes clear that it isn't just any scraping but scraping that results in misappropriation that's to be prevented.
Morgan, I'm curious as to what specifically they asked you to do to the site to prevent it from being "scraped"? One of the more interesting aspects of this, in my view, is that scraping cannot really be prevented if you put the information on the open web, which, as you pointed out above, is the purpose of IDX. However, we can know that Google and other search engines are not misappropriating because they will obey a noindex or noarchive request. So, my question is whether they required that you specify noindex and noarchive or something else? If so, this is a classic case of keeping out those willing to obey the rules without in any way keeping out the law breakers.
That's right Michael, well behaved bots like Google will stay out when requested, yet those that are looking to misappropriate send out spiders that ignore the robots.txt file.
From Wikipedia - http://en.wikipedia.org/wiki/Web_scraping
- Web scraping is closely related to Web indexing, which indexes Web content using a bot and is a universal technique adopted by most search engines. In contrast, Web scraping focuses more on the transformation of unstructured Web content, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet.
- If the application is well behaved, adding entries to robots.txt will be adhered to. You can stop Google and other well-behaved bots this way.
- Sometimes bots declare who they are. Well behaved ones do (for example 'googlebot'). They can be blocked on that basis. Unfortunately, malicious bots may declare they are a normal browser.
Paula Henry
Can NAR prevent R.com from being scraped? I think not!
http://www.bloodhoundrealty.com/BloodhoundBlog/?p=8352
http://www.retechulous.com/category/nar-blows/
knox
I'm working on my opinion (and it agrees with most of you). In the mean time Carolyn has posted on my blog.
http://www.ebaspace.com/2009/05/11/mibor-needs-a-clue/
I wrote a guest post on Knox's blog here:
http://www.ebaspace.com/2009/05/11/mibor-needs-a-clue/
I'm also going to do my best to write a quick post gathering up all the posts in one place- there are a lot of them.
Also besides Knox, I've been talking with other SEO friends including Halfdeck about this-like Michael Wurtzer stated - think the focus needs to be on intent - google and other search engines are not missappropriating the data - may be better to move away from terminology of scraping vs. indexing and have them focus on intent. If part of the issue with the committee turns out to be lack of control of the data when it becomes indexed - think then maybe a noarchive meta tag might satisfy that requirement.
Really hoping that this issue gets resolved but if not - gloves are off.
Off topic, but why are some people's links clickable while other's aren't?
I know I'm missing some posts - even from this thread (I'll keep adding to the list). Here's what I hope soon will be a comprehensive list of posts on this:
http://www.itkeepsyourunning.com/mibor-resources/
this isn't even funny... very disturbing. Instead of working their butts off and spending countless hours to show up in indexed searches (NOT scrapings), not to mention the money invested, these haters file a complaint.
They did it to Marc/Sarasota, and now this. Even worse, the dinosaurs that run the boards and truly don't understand (just read CLIFF's repeated generic response to Morgan) are giving these complainants the time of day. This really does need to be nipped in the bud now. We've invested way too much time and money for it to go up in smoke in the hands of another Agent who's unwilling to commit the same.
@Choice - "We've invested way too much time and money for it to go up in smoke in the hands of another Agent who's unwilling to commit the same."
Exactly, and index or no index, we don't have to worry, the same ones that have been unwilling to invest their time up to this point still won't.
It is the same crowd that thinks they can buy relevance and quality content from the snake oil salesmen, except after being burned time and time again they want those that put out the effort not to have any relevance since they can't buy it.
Our sites will still outshine and outrank theirs index or no index. It just seems asinine to not let the most influential and thorough mechanism of indexing of knowledge in the history of mankind not include real estate site content.
Ron Ogden
Working together, we can find a resolution to this wrong-headed policy that will be balanced and will protect everyone's rights--consumers, agents, and information-owners. Without this grass roots effort and effective lobbying of NAR (hah!--we have to lobby our own lobbyist!), we're headed back to the Real Estate of 20 years ago.
Here's my personal take on the Google indexing controversy and my small contribution to the effort:
http://www.dwellutah.com/blog/google-helps-buyers-find-homes.html
Keep up the good work!
We have more information coming that will shed some light as to why the decisions are being made. I have an agent in my office who has found some very interesting fact that will connect some organizations and maybe shed some light. I hope we can get this to you soon.
We did however make a huge policy change in our office that may floor some people.
This is our position and we stand firm http://skyrealtyaustin.com/blog/why-we-no-longer-syndicate-our-listings.html
I think I am glad I was sitting down thru all these posts. Because the more I read of NAR's ignorance or denial the madder I get. So would Yahoo.com, msn.com and all the rest of the Search Engines become scrapers also? Don't they know we pay their bills? Unreal!!
Kyle
What an idiot. "Scraping" is a very well-defined term and it is established fact that search engines do not, BY DEFINITION, scrape content. This has even been ruled upon by the U.S. Supreme Court. NAR doesn't havea leg to stand on. Their assertion that indexing is a form of scraping is baseless and completely false. Period. There is no room for interpretation here. Take it up with the supereme court if you disagree.
This is becoming a big story, check out;
http://www.inman.com/news/2009/05/13/nar-revisit-idx-indexing-policy
I haven't had a chance to sit down a write a blog post about this. I did just notice the Inman news article. Great job Morgan!
Actually. Let me rephrase that. Great job everyone!
Marc, you need to go to the Inman article and the agent genious article and leave comments comparing your trials and tribulations.
It is not over yet. As long as there are large advertisement based revenue companies out there we need to be prepared. A comment on my blog shows the connection of some of the people. Go up a couple comments on this post and see my link. We are winning this battle and thank goodness, but they still have money and power and that is dangerous for us.
We need to prevent these kinds of things from happening in the future
Curtis, I don't see how anyone can say we are winning anything yet. We have no idea what the comittee's intent will be. I am happy that as a brokerage you have decided not to syndicate listings - it is something tha more should do.
I did want to point out this post:
http://www.retechulous.com/2009/05/11/how-to-scrape-property-listing-data-from-realtorcom/
It shows how easy it is to pull an RSS feed from realtor.com and make a scraper site. NAR has no control over what realtor.com or the other non-real estate agent sites do.
Indexed or non-indexed IDX someone intent on scraping data can do so - again, if the issue is the cached version of the site - a noarchive tag can take care of that. I feel like a broken record - I've read most of the posts, but haven't had the energy to comment. Wishing both Jay and Paula luck tomorrow.
Carolyn wrote: "I don't see how anyone can say we are winning anything yet."
My thoughts exactly Carolyn. In fact, I did a search for "Indianapolis Board of Realtors" and none of the blog entries are on the first page of Google! Having search engine authority on this would certainly help the cause.
Joe - the posts have been targeting MIBOR - 2 negatives for that KW and more on the way up - also google MIBOR and NAR - again, not the most competitive of terms but the posts are there.
Thanks for the suggestion - if needed we can add "Indianapolis Board of Realtors" to the negative keyword list along with NAR and National Associaion of Realtors. I can guarantee it won't be pretty.
Hey Joe - added new string of text in body and tags and repinged! Thanks.
Hey guys,
I'm just a little ole redneck farm boy from Mississippi - can you do me a favor and define Oxymoron for me - I am having trouble with the concept! Is this an Oxymoron - http://www.mibor.com/members/nar_ethics.asp
Oh, I found the definition here - http://en.wikipedia.org/wiki/Oxymoron
Cal, you have too much tiime on your hands. You need to come to SE Washington and work for Colleen as she's working too hard.
knox
@Cal
I got a chuckle where it says:
1. MIBOR sponsored Code of Ethics courses. None available at this time.
Joe,
I'm keeping 5 families fed and an office profitable in what I understand to be a very difficult time in real estate. I work at this stuff 10 hours a day - and I still can't keep up with your content production Kid Squidoo! LOL
Carolyn
I agree, people are high fiven on blogs all over. No matter the outcome of this decission, the threat is still there. So we will suite everyday and keep at it!
Morgan:
I'm a little late to this party. I wanted to read everything to make sure I knew what I'm talking about.
Last year it was Marc, this year Paula, next year it might be may one of us.
I just posted a blog on AR that illustrates the difference between Indexing and Scraping for the benefit of MIBOR. (I hope I got it right)
I titled it and tagged it with "Indianapolis Board of Realtors" to work on that keyword.
See it here, link to it if you think it makes sense:
http://activerain.com/blogsview/1075366/indianapolis-board-of-realtors-indexing-is-not-scraping
Paula Henry
I'm sitting here in DC amazingly surprised by all the support I have experienced in the last week. I'm in no way huge in RE groups and don't get around much ( I mean that in the sincerest way)
I just want o say thanks to everyone here at REW - I am new here and yet you have taken my cause and made it your own.
When I get back to normal, whatever normal is, I am going to get busy learning from everyone here and hopefully give a ittle back.
Thanks,
Paula Henry
On the other side of the addressing the MLS Committee as me a bit tired; but more so, extremely thankful for the support I have had and the online friends I have made.
As an update: the MLS Committee is recommending a change in policy stating:
"Participants must protect IDX information from unauthorized uses. This requirement does not prohibit indexing of IDX sites by search engines."
An affirmative vote will provide us the right to access, index and display our data - we are also give the obligation to protect that data form unauthorized uses; let's be vigilant in maintaining the integrity of the IDX data.
Thanks Again!
Have you seen this open letter on Trulie ato Me at Sky Realty
http://www.trulia.com/blog/pierre_calzadilla/2009/05/an_open_letter_to_mr_red
Got get em
This isn't over yet-won't be for me until the cease and disist is lifted in Indy but wanted to share the info that Jay Thompson provided on twitter. The MLS committee approved a change to the language to state that "This requirement does not prohibit the indexing of IDX sites by search engines."
Here's Jay's shot of the language at the committee meeting: http://twitpic.com/55k0z
The full board of directors still needs to approve the change and more importantly this issue still needs to go back to MIBOR. Again, it is a step in the right direction but just know it isn't completely over yet.
I spoke to the MLS committee today at the NAR event in DC. My thoughts afterwards were as follows that I posted at AgentGenius:
Paula, it was good meeting you today and I appreciated the opportunity to speak before the committee. I think I spoke 2x as long as anybody on the issue–haha
From the proposed modification, I don’t think that stops MIBOR from disallowing indexing within their local if they so choose. It just makes them acknowledge that NAR does not require it all or condone it. So if they continue to disallow the IDX from being indexed and available the search engines they are doing so based on THEIR OWN IDEAS, PREDISPOSITIONS AND BIASES—not those of NAR. So now they own their own policy essentially.
Hopefully after learning that other boards around the country including perhaps one of the largest–MRIS–allow brokers/homeowners to opt out of being included in the indexable IDX on the front end they will rewrite their policies to be more consumer and agent friendly. the default should be opt in inclusion in the IDX which is allowed to be indexed by search engines for efficiency in our 21st century world. If somebody such as the broker Mike Woods who filed a complaint against you who was on record trying to buy the same website/IDX solution as you for those same purposes for which he filed a complaint later–he could opt out with his listings. Utter hypocrisy and backstabbing of competition when he probably decided he did not want to pay up. I can’t wait to see some posts on that situation with keywords for him and his brokerage. Sellers should know about his stance should they not? As a consumer advocate I think so.
Of course I’d like to be a fly on the wall for Mike Woods of msWoods Real Estate, LLC when he tells his sellers, “Don’t worry. I’ve filed a complaint against my competition so that as few people as possible will have this info on your home for sale and google will not index it making it incredibly easy for buyer to find. It’s the last I can do for, Mr. Seller. I put your needs first.”
So let’s see what MIBOR does now that they have to own their discrimination against google indexing IDX info for Sellers to get more exposure. Let’s see if they can look at the issue freshly instead of making this about their egos and defending the old stance on an old NAR policy that is now reworded. This will take courage and humility–both of which are noble but difficult. I believe MIBOR will rise to the task though in the interests of its members and its national reputation. Otherwise this will grow bigger with them in the middle defending the old and indefensible that originated with a jealous broker….
jay--not the phoenix real estate guy! The Jay of Arlington Virgina homes for sale Jay--haha
[edit as you wish ]
A new blog post, please go read, comment, and vote in the poll in the right hand column:
http://www.realestateindustrywatch.com/e0-editorial-and-opinion/
Paula Henry
Cal - Thanks for your support and great post!
Jay - We met with MIBOR last week and they said they could have a "virtual" vote of the MLS Committee to allow the language to be changed for brokers to opt in/out of indexing. It could be voted on as early as June or as late as August, IF it becomes a recommendation of the committee. Not sure how the vote will go. We haven't heard anything yet. They will NOT change their position on the current language.
THANKS "thataway61". I voted.
Only 11 votes so far... come on people, cut and paste the link http://www.realestateindustrywatch.com/e0-editorial-and-opinion/ and vote in the right margin.
http://www.realestateindustrywatch.com/e0-editorial-and-opinion/
Is MIBOR a privately owned MLS? I'm in the Virginia Beach/Hampton Roads, Virginia market and have been dealing with an issue just as great as this for the past three years. Speaking from a strictly market based approach that is. Stay with me on this...many of you have had the wonderful opportunity to employ Morgan/REW for the development of a custom IDX solution or a few of the reputable others in the business. Locally, in our market, Realtors/Agents are not even given the opportunity to purchase an IDX feed, be it through FTP or RETS. Why? Let's just say all the powerhouse brokers own my MLS. I pay for access, but I can't actually access the information outside of the protected shell, without first, becoming a Broker. I would have loved the chance to swing up to DC and participate on a more personal level regarding this National policy. Hasn’t DOJ already come to a conclusion a few months ago? How in the world was this overlooked?
I have to admit, I've been busy consulting and starting a video production company for real estate! and more recently, passing my Broker courses so that I may one day be able to take a well established site of mine and add an indexable IDX feed (of course, coupled with the DOJ mandated VOW policy's...which, oh by the way my MLS doesn't have to abide by). I've driven as deep as possible into our local board, protesting and discussing the ramifications locally, but I still can't give my 15,000 annual Google searchers what they all want! To sum it all up, I'm glad Morgan is in the middle of this. I would also love to become attached at the hip. To save the utter embarrassment of my real estate website that I haven't touched in over a year and a half (hopefully it's not linked in this post, but if it is, I'll have my Broker's license by the end of August and web development has been under way for a while) please feel free to contact me at the below address. If anyone would like to include me on this issue further, please email me at Revolution at VideoAFX dot com I just don’t have the time to follow the original blogosphere anymore. However, if 9 Realtors are discussing this issue face to face and all have located at one central location in the future, please…I will fly in and be number 10. Credibility: NY Real Estate License Virginia Real Estate License 2 servers HTML, JAVA, Flash, php, SEO (White & Black Hat), etc., etc. Technically well spoken The last thing I want this post to be interpreted as is self promotion. I’m (as we speak) investing more money, yet again into my future real estate career. I’m ready, willing, and able to devote my time for the greater good of this entire industry. As probably already noted, this is a very passionate issue of mine. Cell: 757-630-4547REW Blogs User Stats
Currently Online Users:
0
Total Users:
2,389
Entries:
7,601
Unique Views:
6,437,078
Total Views:
6,774,456
Total Comments:
9,371
Total REW Points:
510
As the creative mind behind REW Blogs and the director of technology for Real Estate Webmasters, I am here to help you make the most of our platform.
I am happy to teach all who are willing to learn and look forward to building many great friendships along the way.
I do ask that before immersing yourself in the activities that will no doubt dramatically increase your online exposure such as blogging, creating static pages, adding listings and the like that you take the time to read our terms of service and the rules of blogging here at REW.
We are much different than most social media / blogging platforms in that we are a fully staffed and moderated system. Read More
Contact Me
RSS Feed- This User's Stats
-
Blog Entries: 187Average Blog Rating: 5Unique Views: 438,103Total Views: 479,703Comments Posted: 1,237Comments Received: 3,351REW Points: 240
- Friends
-
Top Rated
REW Blogs RSS Feed
I wonder if this policy were to spread nation wide, how this would affect Zillow, Trulia, etc. Just today we received a listing for a property in Canyon Ridge. I did a search for 'canyon ridge richland wa' and noticed Zillow, realestate.com, etc., ranking for their property listings.
My guess is, if real estate agents were not able to index their idx feeds then these power houses would get even more powerful. As they (Zillow, etc.) would be the only websites indexing agent's listing.